1 [2024-07-17 10:43:48.899915] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010E8-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 2 [2024-07-17 10:43:48.932901] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1623/0, request doing=0/0) 3 [2024-07-17 10:43:48.932953] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=3/0, request done=1623/0, request doing=0/0) 4 [2024-07-17 10:43:48.933204] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=3/0, request done=1623/0, request doing=0/0) 5 [2024-07-17 10:43:48.933543] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=3/0, request done=1623/0, request doing=0/0) 6 [2024-07-17 10:43:48.934348] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 7 [2024-07-17 10:43:48.934357] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 8 [2024-07-17 10:43:48.935184] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 9 [2024-07-17 10:43:48.935348] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 10 [2024-07-17 10:43:48.948875] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 11 [2024-07-17 10:43:48.948893] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010E7-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600874-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010E8-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 12 [2024-07-17 10:43:48.948928] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 13 [2024-07-17 10:43:48.948933] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010E7-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600874-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010E8-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 14 [2024-07-17 10:43:49.049020] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 15 [2024-07-17 10:43:49.049054] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 16 [2024-07-17 10:43:49.066393] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010E9-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 17 [2024-07-17 10:43:49.066421] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010E9-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 18 [2024-07-17 10:43:49.082408] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600875-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 19 [2024-07-17 10:43:49.082434] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600875-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 20 [2024-07-17 10:43:49.099599] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010EA-0-0] [lt=19] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 21 [2024-07-17 10:43:49.099621] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010EA-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 22 [2024-07-17 10:43:49.149140] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 23 [2024-07-17 10:43:49.149159] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010E9-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600875-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EA-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 24 [2024-07-17 10:43:49.149191] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 25 [2024-07-17 10:43:49.149197] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010E9-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600875-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EA-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 26 [2024-07-17 10:43:49.249285] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 27 [2024-07-17 10:43:49.249341] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=34] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 28 [2024-07-17 10:43:49.267418] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010EB-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 29 [2024-07-17 10:43:49.267449] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010EB-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 30 [2024-07-17 10:43:49.283357] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600876-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 31 [2024-07-17 10:43:49.283386] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600876-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 32 [2024-07-17 10:43:49.300475] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010EC-0-0] [lt=20] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 33 [2024-07-17 10:43:49.300498] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010EC-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 34 [2024-07-17 10:43:49.349411] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 35 [2024-07-17 10:43:49.349431] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EB-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600876-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EC-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 36 [2024-07-17 10:43:49.349461] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 37 [2024-07-17 10:43:49.349473] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=12][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EB-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600876-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EC-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 38 [2024-07-17 10:43:49.449577] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 39 [2024-07-17 10:43:49.449628] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=30] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 40 [2024-07-17 10:43:49.467364] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010ED-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 41 [2024-07-17 10:43:49.467401] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010ED-0-0] [lt=34] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 42 [2024-07-17 10:43:49.483542] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600877-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 43 [2024-07-17 10:43:49.483574] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600877-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 44 [2024-07-17 10:43:49.500807] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010EE-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 45 [2024-07-17 10:43:49.500828] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010EE-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 46 [2024-07-17 10:43:49.549711] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 47 [2024-07-17 10:43:49.549731] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010ED-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600877-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EE-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 48 [2024-07-17 10:43:49.549762] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 49 [2024-07-17 10:43:49.549768] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010ED-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600877-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EE-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 50 [2024-07-17 10:43:49.622905] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 51 [2024-07-17 10:43:49.623010] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 52 [2024-07-17 10:43:49.623069] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 53 [2024-07-17 10:43:49.623836] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 54 [2024-07-17 10:43:49.649866] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 55 [2024-07-17 10:43:49.649902] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 56 [2024-07-17 10:43:49.667625] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010EF-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 57 [2024-07-17 10:43:49.667657] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010EF-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 58 [2024-07-17 10:43:49.684006] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600878-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 59 [2024-07-17 10:43:49.684029] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600878-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 60 [2024-07-17 10:43:49.701353] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F0-0-0] [lt=21] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 61 [2024-07-17 10:43:49.701371] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F0-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 62 [2024-07-17 10:43:49.750004] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 63 [2024-07-17 10:43:49.750026] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EF-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600878-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F0-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 64 [2024-07-17 10:43:49.750075] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=48][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 65 [2024-07-17 10:43:49.750087] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=11][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010EF-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600878-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F0-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 66 [2024-07-17 10:43:49.845249] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 67 [2024-07-17 10:43:49.845279] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 68 [2024-07-17 10:43:49.845304] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 69 [2024-07-17 10:43:49.845370] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 70 [2024-07-17 10:43:49.850167] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 71 [2024-07-17 10:43:49.850201] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 72 [2024-07-17 10:43:49.868063] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F1-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 73 [2024-07-17 10:43:49.868098] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F1-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 74 [2024-07-17 10:43:49.884206] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600879-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 75 [2024-07-17 10:43:49.884236] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600879-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 76 [2024-07-17 10:43:49.893738] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1721184229893725, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0 77 [2024-07-17 10:43:49.901722] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F2-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 78 [2024-07-17 10:43:49.901761] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F2-0-0] [lt=36] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 79 [2024-07-17 10:43:49.902179] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1721184229902167, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0 80 [2024-07-17 10:43:49.932822] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=3/0, request done=1627/0, request doing=0/0) 81 [2024-07-17 10:43:49.933154] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1627/0, request doing=0/0) 82 [2024-07-17 10:43:49.933277] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1627/0, request doing=0/0) 83 [2024-07-17 10:43:49.933522] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1626/0, request doing=0/0) 84 [2024-07-17 10:43:49.934222] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 85 [2024-07-17 10:43:49.934355] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 86 [2024-07-17 10:43:49.934375] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 87 [2024-07-17 10:43:49.935187] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 88 [2024-07-17 10:43:49.950286] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 89 [2024-07-17 10:43:49.950307] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F1-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600879-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F2-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 90 [2024-07-17 10:43:49.950341] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=33][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 91 [2024-07-17 10:43:49.950347] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F1-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600879-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F2-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 92 [2024-07-17 10:43:50.050441] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 93 [2024-07-17 10:43:50.050481] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 94 [2024-07-17 10:43:50.068523] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F3-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 95 [2024-07-17 10:43:50.068558] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F3-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 96 [2024-07-17 10:43:50.084685] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060087A-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 97 [2024-07-17 10:43:50.084711] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060087A-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 98 [2024-07-17 10:43:50.101993] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F4-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 99 [2024-07-17 10:43:50.102021] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F4-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 100 [2024-07-17 10:43:50.150568] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 101 [2024-07-17 10:43:50.150589] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F3-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087A-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F4-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 102 [2024-07-17 10:43:50.150618] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=28][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 103 [2024-07-17 10:43:50.150624] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F3-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087A-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F4-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 104 [2024-07-17 10:43:50.250718] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 105 [2024-07-17 10:43:50.250766] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=31] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 106 [2024-07-17 10:43:50.268304] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F5-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 107 [2024-07-17 10:43:50.268343] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F5-0-0] [lt=35] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 108 [2024-07-17 10:43:50.284592] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060087B-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 109 [2024-07-17 10:43:50.284612] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060087B-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 110 [2024-07-17 10:43:50.301776] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F6-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 111 [2024-07-17 10:43:50.301791] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F6-0-0] [lt=15] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 112 [2024-07-17 10:43:50.350851] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 113 [2024-07-17 10:43:50.350870] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F5-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087B-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F6-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 114 [2024-07-17 10:43:50.350899] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=28][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 115 [2024-07-17 10:43:50.350904] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F5-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087B-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F6-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 116 [2024-07-17 10:43:50.450992] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 117 [2024-07-17 10:43:50.451035] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 118 [2024-07-17 10:43:50.468850] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F7-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 119 [2024-07-17 10:43:50.468886] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F7-0-0] [lt=34] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 120 [2024-07-17 10:43:50.485432] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060087C-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 121 [2024-07-17 10:43:50.485454] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060087C-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 122 [2024-07-17 10:43:50.502970] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F8-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 123 [2024-07-17 10:43:50.502987] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F8-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 124 [2024-07-17 10:43:50.551121] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 125 [2024-07-17 10:43:50.551139] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F7-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087C-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F8-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 126 [2024-07-17 10:43:50.551172] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=32][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 127 [2024-07-17 10:43:50.551177] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F7-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087C-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F8-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 128 [2024-07-17 10:43:50.623245] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 129 [2024-07-17 10:43:50.623368] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 130 [2024-07-17 10:43:50.623796] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 131 [2024-07-17 10:43:50.623796] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 132 [2024-07-17 10:43:50.651260] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 133 [2024-07-17 10:43:50.651292] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 134 [2024-07-17 10:43:50.668827] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010F9-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 135 [2024-07-17 10:43:50.668855] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010F9-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 136 [2024-07-17 10:43:50.685115] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060087D-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 137 [2024-07-17 10:43:50.685138] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060087D-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 138 [2024-07-17 10:43:50.702568] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010FA-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 139 [2024-07-17 10:43:50.702591] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010FA-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 140 [2024-07-17 10:43:50.751373] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 141 [2024-07-17 10:43:50.751391] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F9-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087D-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FA-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 142 [2024-07-17 10:43:50.751416] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 143 [2024-07-17 10:43:50.751422] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010F9-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087D-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FA-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 144 [2024-07-17 10:43:50.844602] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 145 [2024-07-17 10:43:50.844900] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 146 [2024-07-17 10:43:50.845310] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 147 [2024-07-17 10:43:50.846402] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 148 [2024-07-17 10:43:50.851503] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 149 [2024-07-17 10:43:50.851533] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 150 [2024-07-17 10:43:50.869311] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010FB-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 151 [2024-07-17 10:43:50.869347] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010FB-0-0] [lt=34] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 152 [2024-07-17 10:43:50.885174] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060087E-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 153 [2024-07-17 10:43:50.885204] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060087E-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 154 [2024-07-17 10:43:50.901346] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1721184230901338, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 155 [2024-07-17 10:43:50.902701] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010FC-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 156 [2024-07-17 10:43:50.902721] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010FC-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 157 [2024-07-17 10:43:50.909785] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1721184230909780, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 158 [2024-07-17 10:43:50.932762] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=3/0, request done=1631/0, request doing=0/0) 159 [2024-07-17 10:43:50.932817] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=3/0, request done=1630/0, request doing=0/0) 160 [2024-07-17 10:43:50.933223] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=3/0, request done=1631/0, request doing=0/0) 161 [2024-07-17 10:43:50.933385] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=3/0, request done=1630/0, request doing=0/0) 162 [2024-07-17 10:43:50.933877] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 163 [2024-07-17 10:43:50.934356] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 164 [2024-07-17 10:43:50.934835] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 165 [2024-07-17 10:43:50.935128] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 166 [2024-07-17 10:43:50.951628] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 167 [2024-07-17 10:43:50.951647] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FB-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087E-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FC-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 168 [2024-07-17 10:43:50.951678] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 169 [2024-07-17 10:43:50.951683] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FB-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087E-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FC-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 170 [2024-07-17 10:43:51.051769] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 171 [2024-07-17 10:43:51.051811] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 172 [2024-07-17 10:43:51.069258] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010FD-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 173 [2024-07-17 10:43:51.069288] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010FD-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 174 [2024-07-17 10:43:51.088466] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060087F-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 175 [2024-07-17 10:43:51.088495] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060087F-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 176 [2024-07-17 10:43:51.105755] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010FE-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 177 [2024-07-17 10:43:51.105777] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010FE-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 178 [2024-07-17 10:43:51.151896] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 179 [2024-07-17 10:43:51.151913] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FD-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087F-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FE-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 180 [2024-07-17 10:43:51.151945] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 181 [2024-07-17 10:43:51.151950] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FD-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060087F-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FE-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 182 [2024-07-17 10:43:51.252031] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 183 [2024-07-17 10:43:51.252077] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=29] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 184 [2024-07-17 10:43:51.269752] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-00000000005010FF-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 185 [2024-07-17 10:43:51.269776] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-00000000005010FF-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 186 [2024-07-17 10:43:51.286056] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600880-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 187 [2024-07-17 10:43:51.286084] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600880-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 188 [2024-07-17 10:43:51.303613] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501100-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 189 [2024-07-17 10:43:51.303630] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501100-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 190 [2024-07-17 10:43:51.352168] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 191 [2024-07-17 10:43:51.352184] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FF-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600880-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501100-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 192 [2024-07-17 10:43:51.352229] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=44][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 193 [2024-07-17 10:43:51.352234] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000005010FF-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600880-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501100-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 194 [2024-07-17 10:43:51.452335] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 195 [2024-07-17 10:43:51.452376] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 196 [2024-07-17 10:43:51.470388] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501101-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 197 [2024-07-17 10:43:51.470412] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501101-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 198 [2024-07-17 10:43:51.486186] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600881-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 199 [2024-07-17 10:43:51.486222] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600881-0-0] [lt=35] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 200 [2024-07-17 10:43:51.503586] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501102-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 201 [2024-07-17 10:43:51.503609] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501102-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 202 [2024-07-17 10:43:51.552463] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 203 [2024-07-17 10:43:51.552480] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501101-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600881-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501102-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 204 [2024-07-17 10:43:51.552511] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 205 [2024-07-17 10:43:51.552521] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=9][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501101-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600881-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501102-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 206 [2024-07-17 10:43:51.623378] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 207 [2024-07-17 10:43:51.623444] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 208 [2024-07-17 10:43:51.623707] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 209 [2024-07-17 10:43:51.623816] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 210 [2024-07-17 10:43:51.652603] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 211 [2024-07-17 10:43:51.652633] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=15] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 212 [2024-07-17 10:43:51.670498] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501103-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 213 [2024-07-17 10:43:51.670524] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501103-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 214 [2024-07-17 10:43:51.686465] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600882-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 215 [2024-07-17 10:43:51.686490] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600882-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 216 [2024-07-17 10:43:51.703909] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501104-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 217 [2024-07-17 10:43:51.703932] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501104-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 218 [2024-07-17 10:43:51.707115] INFO [COMMON] handle (memory_dump.cpp:468) [44931][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) 219 [2024-07-17 10:43:51.710791] INFO handle (memory_dump.cpp:589) [44931][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] statistics: 220 tenant_cnt: 3, max_chunk_cnt: 524288 221 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 222 1 0 2 2 0 223 500 0 45 95 0 224 500 8 13 1 0 225 500 17 9 1 0 226 500 23 4 4 0 227 cost_time: 3521 228 [2024-07-17 10:43:51.752716] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 229 [2024-07-17 10:43:51.752736] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501103-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600882-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501104-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 230 [2024-07-17 10:43:51.752760] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 231 [2024-07-17 10:43:51.752765] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501103-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600882-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501104-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 232 [2024-07-17 10:43:51.844374] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 233 [2024-07-17 10:43:51.844951] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 234 [2024-07-17 10:43:51.845358] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 235 [2024-07-17 10:43:51.845980] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 236 [2024-07-17 10:43:51.852845] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 237 [2024-07-17 10:43:51.852876] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 238 [2024-07-17 10:43:51.870291] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501105-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 239 [2024-07-17 10:43:51.870318] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501105-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 240 [2024-07-17 10:43:51.886298] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600883-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 241 [2024-07-17 10:43:51.886320] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600883-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 242 [2024-07-17 10:43:51.903591] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501106-0-0] [lt=19] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 243 [2024-07-17 10:43:51.903612] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501106-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 244 [2024-07-17 10:43:51.908961] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1721184231908954, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 245 [2024-07-17 10:43:51.917401] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1721184231917396, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 246 [2024-07-17 10:43:51.932660] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=3/0, request done=1635/0, request doing=0/0) 247 [2024-07-17 10:43:51.933104] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=3/0, request done=1634/0, request doing=0/0) 248 [2024-07-17 10:43:51.933364] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=3/0, request done=1634/0, request doing=0/0) 249 [2024-07-17 10:43:51.933369] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=3/0, request done=1634/0, request doing=0/0) 250 [2024-07-17 10:43:51.933962] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 251 [2024-07-17 10:43:51.934224] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 252 [2024-07-17 10:43:51.934343] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 253 [2024-07-17 10:43:51.934361] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 254 [2024-07-17 10:43:51.952963] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 255 [2024-07-17 10:43:51.952979] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501105-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600883-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501106-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 256 [2024-07-17 10:43:51.953003] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 257 [2024-07-17 10:43:51.953012] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=9][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501105-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600883-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501106-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 258 [2024-07-17 10:43:52.053094] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 259 [2024-07-17 10:43:52.053127] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 260 [2024-07-17 10:43:52.070603] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501107-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 261 [2024-07-17 10:43:52.070630] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501107-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 262 [2024-07-17 10:43:52.087158] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600884-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 263 [2024-07-17 10:43:52.087188] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600884-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 264 [2024-07-17 10:43:52.104549] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501108-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 265 [2024-07-17 10:43:52.104571] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501108-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 266 [2024-07-17 10:43:52.153217] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 267 [2024-07-17 10:43:52.153239] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501107-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600884-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501108-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 268 [2024-07-17 10:43:52.153266] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 269 [2024-07-17 10:43:52.153272] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501107-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600884-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501108-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 270 [2024-07-17 10:43:52.253358] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 271 [2024-07-17 10:43:52.253390] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 272 [2024-07-17 10:43:52.270852] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501109-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 273 [2024-07-17 10:43:52.270879] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501109-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 274 [2024-07-17 10:43:52.286814] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600885-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 275 [2024-07-17 10:43:52.286840] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600885-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 276 [2024-07-17 10:43:52.304282] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050110A-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 277 [2024-07-17 10:43:52.304304] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050110A-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 278 [2024-07-17 10:43:52.353470] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 279 [2024-07-17 10:43:52.353491] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501109-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600885-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110A-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 280 [2024-07-17 10:43:52.353518] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 281 [2024-07-17 10:43:52.353524] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501109-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600885-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 282 [2024-07-17 10:43:52.453608] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 283 [2024-07-17 10:43:52.453643] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 284 [2024-07-17 10:43:52.471556] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050110B-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 285 [2024-07-17 10:43:52.471582] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050110B-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 286 [2024-07-17 10:43:52.487326] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600886-0-0] [lt=21] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 287 [2024-07-17 10:43:52.487358] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600886-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 288 [2024-07-17 10:43:52.504915] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050110C-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 289 [2024-07-17 10:43:52.504936] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050110C-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 290 [2024-07-17 10:43:52.553696] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 291 [2024-07-17 10:43:52.553712] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=15][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600886-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110C-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 292 [2024-07-17 10:43:52.553737] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 293 [2024-07-17 10:43:52.553742] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600886-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 294 [2024-07-17 10:43:52.623795] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 295 [2024-07-17 10:43:52.623990] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 296 [2024-07-17 10:43:52.624342] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 297 [2024-07-17 10:43:52.624352] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 298 [2024-07-17 10:43:52.653832] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 299 [2024-07-17 10:43:52.653863] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 300 [2024-07-17 10:43:52.671262] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050110D-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 301 [2024-07-17 10:43:52.671285] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050110D-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 302 [2024-07-17 10:43:52.687321] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600887-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 303 [2024-07-17 10:43:52.687361] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600887-0-0] [lt=38] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 304 [2024-07-17 10:43:52.704638] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050110E-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 305 [2024-07-17 10:43:52.704658] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050110E-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 306 [2024-07-17 10:43:52.753952] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 307 [2024-07-17 10:43:52.753966] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=14][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600887-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110E-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 308 [2024-07-17 10:43:52.753993] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 309 [2024-07-17 10:43:52.753998] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600887-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 310 [2024-07-17 10:43:52.844842] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 311 [2024-07-17 10:43:52.844972] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 312 [2024-07-17 10:43:52.845342] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 313 [2024-07-17 10:43:52.845345] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 314 [2024-07-17 10:43:52.854085] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 315 [2024-07-17 10:43:52.854120] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 316 [2024-07-17 10:43:52.871932] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050110F-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 317 [2024-07-17 10:43:52.871953] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050110F-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 318 [2024-07-17 10:43:52.888104] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600888-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 319 [2024-07-17 10:43:52.888129] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600888-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 320 [2024-07-17 10:43:52.905661] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501110-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 321 [2024-07-17 10:43:52.905682] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501110-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 322 [2024-07-17 10:43:52.916573] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1721184232916567, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 323 [2024-07-17 10:43:52.925012] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=18] PNIO [ratelimit] time: 1721184232925010, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 324 [2024-07-17 10:43:52.932716] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=3/0, request done=1638/0, request doing=0/0) 325 [2024-07-17 10:43:52.932873] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1638/0, request doing=0/0) 326 [2024-07-17 10:43:52.932996] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=3/0, request done=1638/0, request doing=0/0) 327 [2024-07-17 10:43:52.933156] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=1638/0, request doing=0/0) 328 [2024-07-17 10:43:52.934156] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 329 [2024-07-17 10:43:52.934342] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 330 [2024-07-17 10:43:52.934566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 331 [2024-07-17 10:43:52.934984] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 332 [2024-07-17 10:43:52.954206] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 333 [2024-07-17 10:43:52.954218] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=12][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600888-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501110-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 334 [2024-07-17 10:43:52.954241] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 335 [2024-07-17 10:43:52.954246] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050110F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600888-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501110-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 336 [2024-07-17 10:43:53.054336] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 337 [2024-07-17 10:43:53.054372] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 338 [2024-07-17 10:43:53.071760] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501111-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 339 [2024-07-17 10:43:53.071790] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501111-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 340 [2024-07-17 10:43:53.087847] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600889-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 341 [2024-07-17 10:43:53.087874] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600889-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 342 [2024-07-17 10:43:53.105125] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501112-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 343 [2024-07-17 10:43:53.105145] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501112-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 344 [2024-07-17 10:43:53.154458] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 345 [2024-07-17 10:43:53.154482] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501111-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600889-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501112-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 346 [2024-07-17 10:43:53.154512] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 347 [2024-07-17 10:43:53.154518] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501111-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600889-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501112-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 348 [2024-07-17 10:43:53.254605] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 349 [2024-07-17 10:43:53.254658] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=37] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 350 [2024-07-17 10:43:53.272620] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501113-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 351 [2024-07-17 10:43:53.272654] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501113-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 352 [2024-07-17 10:43:53.288852] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060088A-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 353 [2024-07-17 10:43:53.288881] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060088A-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 354 [2024-07-17 10:43:53.306268] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501114-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 355 [2024-07-17 10:43:53.306289] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501114-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 356 [2024-07-17 10:43:53.354753] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 357 [2024-07-17 10:43:53.354780] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501113-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088A-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501114-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 358 [2024-07-17 10:43:53.354808] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 359 [2024-07-17 10:43:53.354813] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501113-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088A-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501114-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 360 [2024-07-17 10:43:53.427430] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=19] [STAT] [ALL_ZONE] INDEX=0/3 ZONE=az1 REGION=BJ STORAGE_TYPE=LOCAL VERSION=88 361 [2024-07-17 10:43:53.427457] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=23] [STAT] [ALL_ZONE] INDEX=1/3 ZONE=az2 REGION=BJ STORAGE_TYPE=LOCAL VERSION=88 362 [2024-07-17 10:43:53.427462] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_ZONE] INDEX=2/3 ZONE=az4 REGION=BJ STORAGE_TYPE=LOCAL VERSION=88 363 [2024-07-17 10:43:53.427468] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_ZONE] INDEX=0/3 ZONE=az1 ZONE_TYPE=ReadWrite VERSION=88 364 [2024-07-17 10:43:53.427472] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] INDEX=1/3 ZONE=az2 ZONE_TYPE=ReadWrite VERSION=88 365 [2024-07-17 10:43:53.427475] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] INDEX=2/3 ZONE=az4 ZONE_TYPE=ReadWrite VERSION=88 366 [2024-07-17 10:43:53.427483] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:454) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=8] [STAT] [ALL_ZONE] COUNT=3 VERSION=88 367 [2024-07-17 10:43:53.427492] INFO [TLOG] purge_stale_zone_records_ (ob_log_all_svr_cache.cpp:660) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] [PURGE] PURGE_COUNT=0 CUR_COUNT=3 VERSION=88 368 [2024-07-17 10:43:53.431551] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=6] [STAT] [ALL_SERVER_LIST] INDEX=0/60 SERVER_ID=33 SERVER="10.16.34.225:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 369 [2024-07-17 10:43:53.431577] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=22] [STAT] [ALL_SERVER_LIST] INDEX=1/60 SERVER_ID=39 SERVER="10.16.4.158:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 370 [2024-07-17 10:43:53.431584] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_SERVER_LIST] INDEX=2/60 SERVER_ID=84 SERVER="10.16.4.182:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 371 [2024-07-17 10:43:53.431590] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=3/60 SERVER_ID=4 SERVER="10.16.52.121:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 372 [2024-07-17 10:43:53.431596] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=4/60 SERVER_ID=73 SERVER="10.16.52.97:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 373 [2024-07-17 10:43:53.431601] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=5/60 SERVER_ID=82 SERVER="10.16.7.30:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 374 [2024-07-17 10:43:53.431606] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=6/60 SERVER_ID=75 SERVER="10.16.8.119:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 375 [2024-07-17 10:43:53.431610] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=7/60 SERVER_ID=31 SERVER="10.16.8.159:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 376 [2024-07-17 10:43:53.431617] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=8/60 SERVER_ID=19 SERVER="10.16.8.94:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 377 [2024-07-17 10:43:53.431621] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=9/60 SERVER_ID=8 SERVER="10.17.11.157:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 378 [2024-07-17 10:43:53.431626] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=10/60 SERVER_ID=21 SERVER="10.17.11.224:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 379 [2024-07-17 10:43:53.431630] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=11/60 SERVER_ID=35 SERVER="10.17.11.225:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 380 [2024-07-17 10:43:53.431634] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=12/60 SERVER_ID=17 SERVER="10.17.11.248:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 381 [2024-07-17 10:43:53.431645] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=9] [STAT] [ALL_SERVER_LIST] INDEX=13/60 SERVER_ID=25 SERVER="10.17.16.224:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 382 [2024-07-17 10:43:53.431651] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_SERVER_LIST] INDEX=14/60 SERVER_ID=24 SERVER="10.17.16.225:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 383 [2024-07-17 10:43:53.431655] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=15/60 SERVER_ID=76 SERVER="10.17.16.249:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 384 [2024-07-17 10:43:53.431659] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=16/60 SERVER_ID=38 SERVER="10.17.19.120:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 385 [2024-07-17 10:43:53.431663] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=17/60 SERVER_ID=9 SERVER="10.17.19.121:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 386 [2024-07-17 10:43:53.431667] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=18/60 SERVER_ID=26 SERVER="10.17.19.96:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 387 [2024-07-17 10:43:53.431672] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=19/60 SERVER_ID=34 SERVER="10.17.2.161:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 388 [2024-07-17 10:43:53.431676] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=20/60 SERVER_ID=74 SERVER="10.17.2.184:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 389 [2024-07-17 10:43:53.431680] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=21/60 SERVER_ID=69 SERVER="10.17.40.32:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 390 [2024-07-17 10:43:53.431684] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=22/60 SERVER_ID=16 SERVER="10.17.40.56:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 391 [2024-07-17 10:43:53.431688] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=23/60 SERVER_ID=1 SERVER="10.17.42.249:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 392 [2024-07-17 10:43:53.431692] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=24/60 SERVER_ID=18 SERVER="10.17.43.31:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 393 [2024-07-17 10:43:53.431696] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=25/60 SERVER_ID=83 SERVER="10.17.43.32:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 394 [2024-07-17 10:43:53.431700] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=26/60 SERVER_ID=3 SERVER="10.17.43.33:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 395 [2024-07-17 10:43:53.431704] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=27/60 SERVER_ID=11 SERVER="10.17.43.36:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 396 [2024-07-17 10:43:53.431710] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_SERVER_LIST] INDEX=28/60 SERVER_ID=78 SERVER="10.17.43.56:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 397 [2024-07-17 10:43:53.431715] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=29/60 SERVER_ID=65 SERVER="10.17.43.57:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 398 [2024-07-17 10:43:53.431719] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=30/60 SERVER_ID=68 SERVER="10.17.57.224:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 399 [2024-07-17 10:43:53.431723] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=31/60 SERVER_ID=37 SERVER="10.17.57.225:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 400 [2024-07-17 10:43:53.431727] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=32/60 SERVER_ID=72 SERVER="10.17.57.248:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 401 [2024-07-17 10:43:53.431731] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=33/60 SERVER_ID=5 SERVER="10.50.54.155:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 402 [2024-07-17 10:43:53.431735] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=34/60 SERVER_ID=29 SERVER="10.50.54.156:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 403 [2024-07-17 10:43:53.431739] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=35/60 SERVER_ID=28 SERVER="10.50.54.180:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 404 [2024-07-17 10:43:53.431743] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=36/60 SERVER_ID=81 SERVER="10.50.54.220:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 405 [2024-07-17 10:43:53.431747] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=37/60 SERVER_ID=22 SERVER="10.50.54.244:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 406 [2024-07-17 10:43:53.431751] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=38/60 SERVER_ID=30 SERVER="10.50.55.114:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 407 [2024-07-17 10:43:53.431755] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=39/60 SERVER_ID=13 SERVER="10.50.55.116:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 408 [2024-07-17 10:43:53.431760] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=40/60 SERVER_ID=14 SERVER="10.50.55.156:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 409 [2024-07-17 10:43:53.431764] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=41/60 SERVER_ID=10 SERVER="10.50.55.180:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 410 [2024-07-17 10:43:53.431768] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=42/60 SERVER_ID=7 SERVER="10.50.55.29:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 411 [2024-07-17 10:43:53.431773] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=43/60 SERVER_ID=2 SERVER="10.50.55.52:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 412 [2024-07-17 10:43:53.431778] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=44/60 SERVER_ID=71 SERVER="10.50.55.90:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 413 [2024-07-17 10:43:53.431782] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=45/60 SERVER_ID=15 SERVER="10.50.55.92:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 414 [2024-07-17 10:43:53.431786] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=46/60 SERVER_ID=27 SERVER="10.50.56.26:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 415 [2024-07-17 10:43:53.431801] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=47/60 SERVER_ID=12 SERVER="10.50.56.53:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 416 [2024-07-17 10:43:53.431805] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=48/60 SERVER_ID=85 SERVER="10.50.73.157:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 417 [2024-07-17 10:43:53.431811] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=49/60 SERVER_ID=20 SERVER="10.50.73.158:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 418 [2024-07-17 10:43:53.431815] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=50/60 SERVER_ID=67 SERVER="10.50.73.181:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 419 [2024-07-17 10:43:53.431819] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=51/60 SERVER_ID=77 SERVER="10.50.73.182:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 420 [2024-07-17 10:43:53.431823] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=52/60 SERVER_ID=32 SERVER="10.50.73.221:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 421 [2024-07-17 10:43:53.431831] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=7] [STAT] [ALL_SERVER_LIST] INDEX=53/60 SERVER_ID=79 SERVER="10.50.73.245:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 422 [2024-07-17 10:43:53.431836] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=54/60 SERVER_ID=6 SERVER="10.50.73.246:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 423 [2024-07-17 10:43:53.431840] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=55/60 SERVER_ID=80 SERVER="10.50.73.29:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=88 424 [2024-07-17 10:43:53.431844] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=56/60 SERVER_ID=66 SERVER="10.50.73.30:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 425 [2024-07-17 10:43:53.431848] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=57/60 SERVER_ID=36 SERVER="10.50.73.54:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=88 426 [2024-07-17 10:43:53.431854] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_SERVER_LIST] INDEX=58/60 SERVER_ID=70 SERVER="10.50.73.93:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 427 [2024-07-17 10:43:53.431859] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=59/60 SERVER_ID=23 SERVER="10.50.75.219:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=88 428 [2024-07-17 10:43:53.431862] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:525) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] COUNT=60 VERSION=88 429 [2024-07-17 10:43:53.431884] INFO [TLOG] purge_stale_records_ (ob_log_all_svr_cache.cpp:646) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] [PURGE] PURGE_COUNT=0 CUR_COUNT=60 VERSION=88 430 [2024-07-17 10:43:53.454915] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 431 [2024-07-17 10:43:53.454959] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 432 [2024-07-17 10:43:53.472471] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501115-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 433 [2024-07-17 10:43:53.472502] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501115-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 434 [2024-07-17 10:43:53.493113] INFO [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] add_server_or_update(next_svr_index=0, svr_items=[{server:"10.16.52.121:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}]) 435 [2024-07-17 10:43:53.493150] INFO [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=34] add_server_or_update(next_svr_index=0, svr_items=[{server:"10.16.52.121:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"10.16.8.94:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}]) 436 [2024-07-17 10:43:53.493167] INFO [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=16] add_server_or_update(next_svr_index=0, svr_items=[{server:"10.16.52.121:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"10.16.8.94:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}, {server:"10.17.16.225:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}]) 437 [2024-07-17 10:43:53.493182] INFO [TLOG] get_ls_svr_list_ (ob_log_route_service.cpp:817) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=10] get_ls_svr_list_ done(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}, svr_list={next_svr_index:0, svr_num:3, svr_items:[{server:"10.16.52.121:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"10.17.16.225:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"10.16.8.94:2882", ranges:[LSN:{94282199040, 94453832809}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}]}, tg_id_=259) 438 [2024-07-17 10:43:53.498366] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060088B-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 439 [2024-07-17 10:43:53.498399] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060088B-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 440 [2024-07-17 10:43:53.515935] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501116-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 441 [2024-07-17 10:43:53.515957] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501116-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 442 [2024-07-17 10:43:53.555042] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 443 [2024-07-17 10:43:53.555066] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501115-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088B-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501116-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 444 [2024-07-17 10:43:53.555100] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 445 [2024-07-17 10:43:53.555106] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501115-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088B-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501116-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 446 [2024-07-17 10:43:53.623285] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 447 [2024-07-17 10:43:53.623360] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 448 [2024-07-17 10:43:53.623672] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 449 [2024-07-17 10:43:53.624366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 450 [2024-07-17 10:43:53.655190] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 451 [2024-07-17 10:43:53.655227] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 452 [2024-07-17 10:43:53.672679] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501117-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 453 [2024-07-17 10:43:53.672705] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501117-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 454 [2024-07-17 10:43:53.689064] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060088C-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 455 [2024-07-17 10:43:53.689090] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060088C-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 456 [2024-07-17 10:43:53.706609] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501118-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 457 [2024-07-17 10:43:53.706632] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501118-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 458 [2024-07-17 10:43:53.755308] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 459 [2024-07-17 10:43:53.755326] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501117-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088C-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501118-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 460 [2024-07-17 10:43:53.755362] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=35][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 461 [2024-07-17 10:43:53.755367] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501117-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088C-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501118-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 462 [2024-07-17 10:43:53.844639] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 463 [2024-07-17 10:43:53.845067] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 464 [2024-07-17 10:43:53.845105] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 465 [2024-07-17 10:43:53.846103] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 466 [2024-07-17 10:43:53.855453] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 467 [2024-07-17 10:43:53.855488] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 468 [2024-07-17 10:43:53.873243] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501119-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 469 [2024-07-17 10:43:53.873281] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501119-0-0] [lt=35] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 470 [2024-07-17 10:43:53.889782] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060088D-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 471 [2024-07-17 10:43:53.889811] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060088D-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 472 [2024-07-17 10:43:53.907162] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050111A-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 473 [2024-07-17 10:43:53.907201] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050111A-0-0] [lt=37] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 474 [2024-07-17 10:43:53.924199] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1721184233924191, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007624, add_bytes: 0 475 [2024-07-17 10:43:53.932635] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1721184233932629, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 476 [2024-07-17 10:43:53.932726] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=3/0, request done=1641/0, request doing=0/0) 477 [2024-07-17 10:43:53.932841] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=1642/0, request doing=0/0) 478 [2024-07-17 10:43:53.933216] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1642/0, request doing=0/0) 479 [2024-07-17 10:43:53.933305] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=3/0, request done=1642/0, request doing=0/0) 480 [2024-07-17 10:43:53.933913] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 481 [2024-07-17 10:43:53.934354] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 482 [2024-07-17 10:43:53.934369] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 483 [2024-07-17 10:43:53.934368] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 484 [2024-07-17 10:43:53.955565] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 485 [2024-07-17 10:43:53.955581] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=15][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501119-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088D-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111A-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 486 [2024-07-17 10:43:53.955609] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 487 [2024-07-17 10:43:53.955614] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501119-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088D-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 488 [2024-07-17 10:43:53.975498] INFO [TLOG.FETCHER] print_stat (ob_log_fetch_stream_container_mgr.cpp:178) [45064][][T0][Y0-0000000000000000-0-0] [lt=19] [STAT] [FS_CONTAINER_POOL] USED=1 FREE=0 FIXED=16 DYNAMIC=0 489 [2024-07-17 10:43:53.975525] INFO [TLOG.FETCHER] print_stat (ob_log_fetch_stream_pool.cpp:76) [45064][][T0][Y0-0000000000000000-0-0] [lt=25] [STAT] [FETCH_STREAM_POOL] USED=0 FREE=0 FIXED=16 DYNAMIC=0 490 [2024-07-17 10:43:53.975532] INFO [TLOG.FETCHER] print_stat (ob_log_fetch_log_rpc.cpp:1400) [45064][][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [RPC_RESULT_POOL] USED=0 FREE=0 FIXED=16 DYNAMIC=0 491 [2024-07-17 10:43:53.975583] INFO [TLOG.FETCHER] print_fetcher_stat_ (ob_log_fetcher.cpp:837) [45064][][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [FETCHER](upper_limit="[2024-07-16 15:15:15.070225]", dml_progress_limit_sec=30, fetcher_delay="[19:29:08.905347]") 492 [2024-07-17 10:43:53.975684] INFO [TLOG.FETCHER] print_k_slowest_ls (ob_log_ls_fetch_mgr.cpp:429) [45064][][T0][Y0-0000000000000000-0-0] [lt=7] [STAT] slow fetch progress start. ls_count=1/1 493 [2024-07-17 10:43:53.975700] INFO [TLOG.FETCHER] print_fetch_progress (ob_log_ls_fetch_ctx.cpp:1454) [45064][][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] slow fetch progress idx=0/1 tls_id={tenant_id:1004, ls_id:{id:1}} mod=IDLE_POOL discarded=0 delay=[19:29:08.905459] tps=0.00 progress={next_lsn:{lsn:18446744073709551615}, log_progress:"[2024-07-16 15:14:45.070225]", log_touch_tstamp:"[2024-07-17 10:36:33.993829]"} 494 [2024-07-17 10:43:53.975708] INFO [TLOG.FETCHER] print_k_slowest_ls (ob_log_ls_fetch_mgr.cpp:436) [45064][][T0][Y0-0000000000000000-0-0] [lt=7] [STAT] slow fetch progress end 495 [2024-07-17 10:43:53.975713] INFO [TLOG.FETCHER] print_k_slowest_ls (ob_log_ls_fetch_mgr.cpp:441) [45064][][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] slow dispatch progress start. ls_count=1/1 496 [2024-07-17 10:43:53.975723] INFO [TLOG.FETCHER] print_dispatch_progress (ob_log_ls_fetch_ctx.cpp:1476) [45064][][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] slow dispatch progress idx=0/1 tls_id={tenant_id:1004, ls_id:{id:1}} delay=[19:29:08.905459] pending_task(queue/total)=0/0 dispatch_progress=[2024-07-16 15:14:45.070225] last_dispatch_log_lsn=18446744073709551615 next_task=INVALID next_trans(log_lsn=18446744073709551615,committed=0,ready_to_commit=0,global_version=[1970-01-01 08:00:00.000000]) checkpoint=[1970-01-01 08:00:00.000000] 497 [2024-07-17 10:43:53.975735] INFO [TLOG.FETCHER] print_k_slowest_ls (ob_log_ls_fetch_mgr.cpp:448) [45064][][T0][Y0-0000000000000000-0-0] [lt=11] [STAT] slow dispatch progress end 498 [2024-07-17 10:43:54.025093] INFO [TLOG] add_ls_and_fetch_until_the_progress_is_reached (ob_log_meta_data_fetcher.cpp:195) [44926][][T0][Y0-0000000000000000-0-0] [lt=6] ObLogFetcher check_progress is less than end_timestamp(tenant_id=1004, cur_progress=1721114085070225349, end_tstamp_ns=1721183088000000000) 499 [2024-07-17 10:43:54.025131] INFO [TLOG] print_stat_info (ob_log_task_pool.h:266) [44926][][T0][Y0-0000000000000000-0-0] [lt=33] [STAT] [TRANS_TASK_POOL] OUT=0/1000 FIXED=0/1000 DYNAMIC=0 PAGE_POOL=1000/1000 500 [2024-07-17 10:43:54.055700] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 501 [2024-07-17 10:43:54.055743] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 502 [2024-07-17 10:43:54.073646] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050111B-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 503 [2024-07-17 10:43:54.073684] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050111B-0-0] [lt=35] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 504 [2024-07-17 10:43:54.089180] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060088E-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 505 [2024-07-17 10:43:54.089208] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060088E-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 506 [2024-07-17 10:43:54.106626] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050111C-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 507 [2024-07-17 10:43:54.106657] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050111C-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 508 [2024-07-17 10:43:54.155834] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 509 [2024-07-17 10:43:54.155857] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088E-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111C-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 510 [2024-07-17 10:43:54.155886] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=28][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 511 [2024-07-17 10:43:54.155891] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088E-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 512 [2024-07-17 10:43:54.255982] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 513 [2024-07-17 10:43:54.256020] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 514 [2024-07-17 10:43:54.273442] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050111D-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 515 [2024-07-17 10:43:54.273473] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050111D-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 516 [2024-07-17 10:43:54.289514] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060088F-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 517 [2024-07-17 10:43:54.289542] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060088F-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 518 [2024-07-17 10:43:54.307083] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050111E-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 519 [2024-07-17 10:43:54.307103] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050111E-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 520 [2024-07-17 10:43:54.356104] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 521 [2024-07-17 10:43:54.356124] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088F-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111E-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 522 [2024-07-17 10:43:54.356151] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 523 [2024-07-17 10:43:54.356156] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060088F-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 524 [2024-07-17 10:43:54.456247] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 525 [2024-07-17 10:43:54.456284] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 526 [2024-07-17 10:43:54.473996] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050111F-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 527 [2024-07-17 10:43:54.474031] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050111F-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 528 [2024-07-17 10:43:54.490426] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600890-0-0] [lt=20] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 529 [2024-07-17 10:43:54.490453] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600890-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 530 [2024-07-17 10:43:54.507873] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501120-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 531 [2024-07-17 10:43:54.507904] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501120-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 532 [2024-07-17 10:43:54.556369] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 533 [2024-07-17 10:43:54.556387] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600890-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501120-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 534 [2024-07-17 10:43:54.556413] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 535 [2024-07-17 10:43:54.556418] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050111F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600890-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501120-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 536 [2024-07-17 10:43:54.623413] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 537 [2024-07-17 10:43:54.623702] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 538 [2024-07-17 10:43:54.623909] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 539 [2024-07-17 10:43:54.624039] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 540 [2024-07-17 10:43:54.656509] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 541 [2024-07-17 10:43:54.656545] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 542 [2024-07-17 10:43:54.674042] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501121-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 543 [2024-07-17 10:43:54.674072] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501121-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 544 [2024-07-17 10:43:54.690702] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600891-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 545 [2024-07-17 10:43:54.690732] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600891-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 546 [2024-07-17 10:43:54.708064] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501122-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 547 [2024-07-17 10:43:54.708089] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501122-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 548 [2024-07-17 10:43:54.756631] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 549 [2024-07-17 10:43:54.756650] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501121-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600891-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501122-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 550 [2024-07-17 10:43:54.756680] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 551 [2024-07-17 10:43:54.756688] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=7][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501121-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600891-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501122-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 552 [2024-07-17 10:43:54.844230] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 553 [2024-07-17 10:43:54.844303] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 554 [2024-07-17 10:43:54.846014] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 555 [2024-07-17 10:43:54.846015] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 556 [2024-07-17 10:43:54.856779] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 557 [2024-07-17 10:43:54.856812] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 558 [2024-07-17 10:43:54.874828] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501123-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 559 [2024-07-17 10:43:54.874859] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501123-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 560 [2024-07-17 10:43:54.890717] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600892-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 561 [2024-07-17 10:43:54.890747] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600892-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 562 [2024-07-17 10:43:54.908009] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501124-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 563 [2024-07-17 10:43:54.908032] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501124-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 564 [2024-07-17 10:43:54.931801] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1721184234931799, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007608, add_bytes: 0 565 [2024-07-17 10:43:54.932779] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=3/0, request done=1646/0, request doing=0/0) 566 [2024-07-17 10:43:54.932897] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=3/0, request done=1645/0, request doing=0/0) 567 [2024-07-17 10:43:54.933065] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=3/0, request done=1646/0, request doing=0/0) 568 [2024-07-17 10:43:54.933147] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1645/0, request doing=0/0) 569 [2024-07-17 10:43:54.934143] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 570 [2024-07-17 10:43:54.934383] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 571 [2024-07-17 10:43:54.934423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 572 [2024-07-17 10:43:54.935071] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 573 [2024-07-17 10:43:54.940245] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1721184234940243, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 574 [2024-07-17 10:43:54.956896] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 575 [2024-07-17 10:43:54.956913] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501123-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600892-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501124-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 576 [2024-07-17 10:43:54.956937] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 577 [2024-07-17 10:43:54.956942] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501123-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600892-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501124-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 578 [2024-07-17 10:43:54.975808] INFO [TLOG.PARSER] stat_ignored_tps (ob_log_cluster_id_filter.cpp:130) [45064][][T0][Y0-0000000000000000-0-0] [lt=4] [TPS_STAT] CLUSTER_ID_IGNORED_PART_TPS=0.000 579 [2024-07-17 10:43:55.057035] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 580 [2024-07-17 10:43:55.057077] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 581 [2024-07-17 10:43:55.074456] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501125-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 582 [2024-07-17 10:43:55.074487] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501125-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 583 [2024-07-17 10:43:55.090233] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600893-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 584 [2024-07-17 10:43:55.090265] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600893-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 585 [2024-07-17 10:43:55.107619] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501126-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 586 [2024-07-17 10:43:55.107647] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501126-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 587 [2024-07-17 10:43:55.157171] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 588 [2024-07-17 10:43:55.157197] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501125-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600893-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501126-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 589 [2024-07-17 10:43:55.157245] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=47][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 590 [2024-07-17 10:43:55.157251] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501125-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600893-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501126-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 591 [2024-07-17 10:43:55.257352] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 592 [2024-07-17 10:43:55.257406] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=38] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 593 [2024-07-17 10:43:55.274980] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501127-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 594 [2024-07-17 10:43:55.275011] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501127-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 595 [2024-07-17 10:43:55.291440] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600894-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 596 [2024-07-17 10:43:55.291473] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600894-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 597 [2024-07-17 10:43:55.309099] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501128-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 598 [2024-07-17 10:43:55.309121] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501128-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 599 [2024-07-17 10:43:55.357497] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 600 [2024-07-17 10:43:55.357523] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501127-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600894-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501128-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 601 [2024-07-17 10:43:55.357551] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 602 [2024-07-17 10:43:55.357556] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501127-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600894-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501128-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 603 [2024-07-17 10:43:55.457649] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 604 [2024-07-17 10:43:55.457702] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=37] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 605 [2024-07-17 10:43:55.475703] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501129-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 606 [2024-07-17 10:43:55.475741] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501129-0-0] [lt=36] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 607 [2024-07-17 10:43:55.492706] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600895-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 608 [2024-07-17 10:43:55.492732] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600895-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 609 [2024-07-17 10:43:55.510201] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050112A-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 610 [2024-07-17 10:43:55.510228] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050112A-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 611 [2024-07-17 10:43:55.557787] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 612 [2024-07-17 10:43:55.557804] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501129-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600895-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112A-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 613 [2024-07-17 10:43:55.557836] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 614 [2024-07-17 10:43:55.557841] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501129-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600895-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 615 [2024-07-17 10:43:55.623455] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 616 [2024-07-17 10:43:55.623486] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 617 [2024-07-17 10:43:55.623750] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 618 [2024-07-17 10:43:55.623960] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 619 [2024-07-17 10:43:55.657929] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 620 [2024-07-17 10:43:55.657957] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 621 [2024-07-17 10:43:55.675961] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050112B-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 622 [2024-07-17 10:43:55.675993] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050112B-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 623 [2024-07-17 10:43:55.691933] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600896-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 624 [2024-07-17 10:43:55.691959] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600896-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 625 [2024-07-17 10:43:55.709527] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050112C-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 626 [2024-07-17 10:43:55.709549] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050112C-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 627 [2024-07-17 10:43:55.758025] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 628 [2024-07-17 10:43:55.758051] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600896-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112C-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 629 [2024-07-17 10:43:55.758080] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=28][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 630 [2024-07-17 10:43:55.758086] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600896-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 631 [2024-07-17 10:43:55.844346] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 632 [2024-07-17 10:43:55.845325] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 633 [2024-07-17 10:43:55.845365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 634 [2024-07-17 10:43:55.845686] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 635 [2024-07-17 10:43:55.858173] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 636 [2024-07-17 10:43:55.858213] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=28] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 637 [2024-07-17 10:43:55.875486] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050112D-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 638 [2024-07-17 10:43:55.875516] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050112D-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 639 [2024-07-17 10:43:55.891769] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600897-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 640 [2024-07-17 10:43:55.891799] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600897-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 641 [2024-07-17 10:43:55.909093] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050112E-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 642 [2024-07-17 10:43:55.909124] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050112E-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 643 [2024-07-17 10:43:55.932627] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=1649/0, request doing=0/0) 644 [2024-07-17 10:43:55.932827] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=3/0, request done=1649/0, request doing=0/0) 645 [2024-07-17 10:43:55.933163] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=3/0, request done=1650/0, request doing=0/0) 646 [2024-07-17 10:43:55.933398] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 647 [2024-07-17 10:43:55.933559] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1649/0, request doing=0/0) 648 [2024-07-17 10:43:55.934342] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 649 [2024-07-17 10:43:55.934361] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 650 [2024-07-17 10:43:55.934495] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 651 [2024-07-17 10:43:55.939424] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1721184235939419, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0 652 [2024-07-17 10:43:55.947863] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1721184235947859, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 653 [2024-07-17 10:43:55.958300] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 654 [2024-07-17 10:43:55.958320] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600897-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112E-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 655 [2024-07-17 10:43:55.958363] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=42][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 656 [2024-07-17 10:43:55.958368] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600897-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 657 [2024-07-17 10:43:56.058461] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 658 [2024-07-17 10:43:56.058498] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 659 [2024-07-17 10:43:56.075940] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050112F-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 660 [2024-07-17 10:43:56.075974] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050112F-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 661 [2024-07-17 10:43:56.092054] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600898-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 662 [2024-07-17 10:43:56.092087] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600898-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 663 [2024-07-17 10:43:56.109524] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501130-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 664 [2024-07-17 10:43:56.109555] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501130-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 665 [2024-07-17 10:43:56.158588] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 666 [2024-07-17 10:43:56.158609] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600898-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501130-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 667 [2024-07-17 10:43:56.158637] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 668 [2024-07-17 10:43:56.158643] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050112F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600898-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501130-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 669 [2024-07-17 10:43:56.258734] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 670 [2024-07-17 10:43:56.258769] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 671 [2024-07-17 10:43:56.276676] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501131-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 672 [2024-07-17 10:43:56.276704] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501131-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 673 [2024-07-17 10:43:56.293235] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-0000000000600899-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 674 [2024-07-17 10:43:56.293264] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-0000000000600899-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 675 [2024-07-17 10:43:56.310844] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501132-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 676 [2024-07-17 10:43:56.310865] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501132-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 677 [2024-07-17 10:43:56.358839] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 678 [2024-07-17 10:43:56.358857] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501131-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600899-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501132-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 679 [2024-07-17 10:43:56.358882] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 680 [2024-07-17 10:43:56.358887] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501131-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000600899-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501132-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 681 [2024-07-17 10:43:56.458971] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 682 [2024-07-17 10:43:56.458997] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 683 [2024-07-17 10:43:56.476890] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501133-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 684 [2024-07-17 10:43:56.476914] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501133-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 685 [2024-07-17 10:43:56.492474] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060089A-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 686 [2024-07-17 10:43:56.492500] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060089A-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 687 [2024-07-17 10:43:56.509825] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501134-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 688 [2024-07-17 10:43:56.509843] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501134-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 689 [2024-07-17 10:43:56.559031] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 690 [2024-07-17 10:43:56.559049] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501133-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089A-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501134-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 691 [2024-07-17 10:43:56.559072] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 692 [2024-07-17 10:43:56.559078] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501133-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089A-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501134-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 693 [2024-07-17 10:43:56.623721] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 694 [2024-07-17 10:43:56.623795] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 695 [2024-07-17 10:43:56.623990] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 696 [2024-07-17 10:43:56.624261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 697 [2024-07-17 10:43:56.659160] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 698 [2024-07-17 10:43:56.659185] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=15] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 699 [2024-07-17 10:43:56.676676] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501135-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 700 [2024-07-17 10:43:56.676703] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501135-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 701 [2024-07-17 10:43:56.694370] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060089B-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 702 [2024-07-17 10:43:56.694400] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060089B-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 703 [2024-07-17 10:43:56.711701] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501136-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 704 [2024-07-17 10:43:56.711720] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501136-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 705 [2024-07-17 10:43:56.759259] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 706 [2024-07-17 10:43:56.759276] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501135-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089B-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501136-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 707 [2024-07-17 10:43:56.759299] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 708 [2024-07-17 10:43:56.759304] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501135-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089B-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501136-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 709 [2024-07-17 10:43:56.844914] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 710 [2024-07-17 10:43:56.845085] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 711 [2024-07-17 10:43:56.845252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 712 [2024-07-17 10:43:56.845482] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 713 [2024-07-17 10:43:56.859400] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 714 [2024-07-17 10:43:56.859423] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=12] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 715 [2024-07-17 10:43:56.876942] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501137-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 716 [2024-07-17 10:43:56.876968] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501137-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 717 [2024-07-17 10:43:56.892858] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060089C-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 718 [2024-07-17 10:43:56.892881] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060089C-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 719 [2024-07-17 10:43:56.910411] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501138-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 720 [2024-07-17 10:43:56.910431] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501138-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 721 [2024-07-17 10:43:56.932787] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=3/0, request done=1653/0, request doing=0/0) 722 [2024-07-17 10:43:56.932910] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=3/0, request done=1653/0, request doing=0/0) 723 [2024-07-17 10:43:56.933004] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=3/0, request done=1653/0, request doing=0/0) 724 [2024-07-17 10:43:56.933468] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=1653/0, request doing=0/0) 725 [2024-07-17 10:43:56.933500] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 726 [2024-07-17 10:43:56.934370] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 727 [2024-07-17 10:43:56.934465] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 728 [2024-07-17 10:43:56.935515] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 729 [2024-07-17 10:43:56.947033] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1721184236947031, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0 730 [2024-07-17 10:43:56.955478] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1721184236955476, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 731 [2024-07-17 10:43:56.959501] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 732 [2024-07-17 10:43:56.959515] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=13][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501137-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089C-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501138-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 733 [2024-07-17 10:43:56.959536] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 734 [2024-07-17 10:43:56.959541] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501137-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089C-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501138-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 735 [2024-07-17 10:43:57.059577] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 736 [2024-07-17 10:43:57.059608] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 737 [2024-07-17 10:43:57.076999] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501139-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 738 [2024-07-17 10:43:57.077024] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501139-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 739 [2024-07-17 10:43:57.092974] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060089D-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 740 [2024-07-17 10:43:57.092999] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060089D-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 741 [2024-07-17 10:43:57.110092] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050113A-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 742 [2024-07-17 10:43:57.110115] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050113A-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 743 [2024-07-17 10:43:57.159690] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 744 [2024-07-17 10:43:57.159708] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501139-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089D-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113A-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 745 [2024-07-17 10:43:57.159732] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 746 [2024-07-17 10:43:57.159737] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501139-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089D-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 747 [2024-07-17 10:43:57.259816] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 748 [2024-07-17 10:43:57.259853] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 749 [2024-07-17 10:43:57.277716] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050113B-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 750 [2024-07-17 10:43:57.277742] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050113B-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 751 [2024-07-17 10:43:57.293851] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060089E-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 752 [2024-07-17 10:43:57.293876] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060089E-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 753 [2024-07-17 10:43:57.312504] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050113C-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 754 [2024-07-17 10:43:57.312529] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050113C-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 755 [2024-07-17 10:43:57.359935] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 756 [2024-07-17 10:43:57.359953] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089E-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113C-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 757 [2024-07-17 10:43:57.359977] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 758 [2024-07-17 10:43:57.359982] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089E-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 759 [2024-07-17 10:43:57.460063] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 760 [2024-07-17 10:43:57.460090] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 761 [2024-07-17 10:43:57.477606] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050113D-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 762 [2024-07-17 10:43:57.477631] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050113D-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 763 [2024-07-17 10:43:57.493789] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-000000000060089F-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 764 [2024-07-17 10:43:57.493813] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-000000000060089F-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 765 [2024-07-17 10:43:57.511064] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050113E-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 766 [2024-07-17 10:43:57.511086] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050113E-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 767 [2024-07-17 10:43:57.560169] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 768 [2024-07-17 10:43:57.560187] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089F-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113E-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 769 [2024-07-17 10:43:57.560210] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 770 [2024-07-17 10:43:57.560216] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000060089F-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 771 [2024-07-17 10:43:57.623351] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 772 [2024-07-17 10:43:57.623374] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 773 [2024-07-17 10:43:57.623749] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 774 [2024-07-17 10:43:57.624024] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 775 [2024-07-17 10:43:57.660293] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 776 [2024-07-17 10:43:57.660313] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=11] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 777 [2024-07-17 10:43:57.677709] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050113F-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 778 [2024-07-17 10:43:57.677735] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050113F-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 779 [2024-07-17 10:43:57.693630] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A0-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 780 [2024-07-17 10:43:57.693654] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A0-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 781 [2024-07-17 10:43:57.711060] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501140-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 782 [2024-07-17 10:43:57.711082] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501140-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 783 [2024-07-17 10:43:57.760396] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 784 [2024-07-17 10:43:57.760411] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=14][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A0-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501140-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 785 [2024-07-17 10:43:57.760432] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 786 [2024-07-17 10:43:57.760437] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050113F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A0-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501140-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 787 [2024-07-17 10:43:57.844352] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 788 [2024-07-17 10:43:57.844636] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 789 [2024-07-17 10:43:57.844870] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 790 [2024-07-17 10:43:57.845115] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 791 [2024-07-17 10:43:57.860524] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 792 [2024-07-17 10:43:57.860555] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=13] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 793 [2024-07-17 10:43:57.878024] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501141-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 794 [2024-07-17 10:43:57.878050] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501141-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 795 [2024-07-17 10:43:57.894183] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A1-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 796 [2024-07-17 10:43:57.894207] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A1-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 797 [2024-07-17 10:43:57.911459] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501142-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 798 [2024-07-17 10:43:57.911486] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501142-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 799 [2024-07-17 10:43:57.933041] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=3/0, request done=1657/0, request doing=0/0) 800 [2024-07-17 10:43:57.933241] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=1657/0, request doing=0/0) 801 [2024-07-17 10:43:57.933242] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=3/0, request done=1656/0, request doing=0/0) 802 [2024-07-17 10:43:57.933517] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=1657/0, request doing=0/0) 803 [2024-07-17 10:43:57.933591] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 804 [2024-07-17 10:43:57.933796] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 805 [2024-07-17 10:43:57.934238] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 806 [2024-07-17 10:43:57.934653] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 807 [2024-07-17 10:43:57.954651] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1721184237954647, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 808 [2024-07-17 10:43:57.960628] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=15][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 809 [2024-07-17 10:43:57.960639] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=10][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501141-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A1-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501142-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 810 [2024-07-17 10:43:57.960669] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 811 [2024-07-17 10:43:57.960675] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501141-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A1-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501142-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 812 [2024-07-17 10:43:57.963093] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1721184237963091, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 813 [2024-07-17 10:43:58.060758] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 814 [2024-07-17 10:43:58.060783] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 815 [2024-07-17 10:43:58.078620] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501143-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 816 [2024-07-17 10:43:58.078647] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501143-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 817 [2024-07-17 10:43:58.094621] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A2-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 818 [2024-07-17 10:43:58.094645] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A2-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 819 [2024-07-17 10:43:58.111857] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501144-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 820 [2024-07-17 10:43:58.111878] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501144-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 821 [2024-07-17 10:43:58.160859] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 822 [2024-07-17 10:43:58.160882] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501143-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A2-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501144-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 823 [2024-07-17 10:43:58.160910] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 824 [2024-07-17 10:43:58.160915] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501143-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A2-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501144-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 825 [2024-07-17 10:43:58.261001] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 826 [2024-07-17 10:43:58.261036] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 827 [2024-07-17 10:43:58.278267] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501145-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 828 [2024-07-17 10:43:58.278315] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501145-0-0] [lt=45] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 829 [2024-07-17 10:43:58.294790] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A3-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 830 [2024-07-17 10:43:58.294837] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A3-0-0] [lt=45] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 831 [2024-07-17 10:43:58.312062] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501146-0-0] [lt=27] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 832 [2024-07-17 10:43:58.312108] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501146-0-0] [lt=42] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 833 [2024-07-17 10:43:58.361100] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 834 [2024-07-17 10:43:58.361131] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=30][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501145-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A3-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501146-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 835 [2024-07-17 10:43:58.361163] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=32][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 836 [2024-07-17 10:43:58.361169] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501145-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A3-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501146-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 837 [2024-07-17 10:43:58.461264] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=29] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 838 [2024-07-17 10:43:58.461301] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 839 [2024-07-17 10:43:58.478736] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501147-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 840 [2024-07-17 10:43:58.478775] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501147-0-0] [lt=36] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 841 [2024-07-17 10:43:58.494922] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A4-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 842 [2024-07-17 10:43:58.494970] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A4-0-0] [lt=46] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 843 [2024-07-17 10:43:58.501619] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=19] [STAT] [ALL_ZONE] INDEX=0/3 ZONE=az1 REGION=BJ STORAGE_TYPE=LOCAL VERSION=89 844 [2024-07-17 10:43:58.501645] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=18] [STAT] [ALL_ZONE] INDEX=1/3 ZONE=az2 REGION=BJ STORAGE_TYPE=LOCAL VERSION=89 845 [2024-07-17 10:43:58.501651] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_ZONE] INDEX=2/3 ZONE=az4 REGION=BJ STORAGE_TYPE=LOCAL VERSION=89 846 [2024-07-17 10:43:58.501656] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] INDEX=0/3 ZONE=az1 ZONE_TYPE=ReadWrite VERSION=89 847 [2024-07-17 10:43:58.501660] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] INDEX=1/3 ZONE=az2 ZONE_TYPE=ReadWrite VERSION=89 848 [2024-07-17 10:43:58.501663] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_ZONE] INDEX=2/3 ZONE=az4 ZONE_TYPE=ReadWrite VERSION=89 849 [2024-07-17 10:43:58.501666] INFO [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:454) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] COUNT=3 VERSION=89 850 [2024-07-17 10:43:58.501675] INFO [TLOG] purge_stale_zone_records_ (ob_log_all_svr_cache.cpp:660) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] [PURGE] PURGE_COUNT=0 CUR_COUNT=3 VERSION=89 851 [2024-07-17 10:43:58.506786] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=0/60 SERVER_ID=33 SERVER="10.16.34.225:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 852 [2024-07-17 10:43:58.506798] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=10] [STAT] [ALL_SERVER_LIST] INDEX=1/60 SERVER_ID=39 SERVER="10.16.4.158:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 853 [2024-07-17 10:43:58.506803] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=2/60 SERVER_ID=84 SERVER="10.16.4.182:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 854 [2024-07-17 10:43:58.506808] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=3/60 SERVER_ID=4 SERVER="10.16.52.121:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 855 [2024-07-17 10:43:58.506812] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=4/60 SERVER_ID=73 SERVER="10.16.52.97:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 856 [2024-07-17 10:43:58.506817] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=5/60 SERVER_ID=82 SERVER="10.16.7.30:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 857 [2024-07-17 10:43:58.506827] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=9] [STAT] [ALL_SERVER_LIST] INDEX=6/60 SERVER_ID=75 SERVER="10.16.8.119:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 858 [2024-07-17 10:43:58.506832] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=7/60 SERVER_ID=31 SERVER="10.16.8.159:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 859 [2024-07-17 10:43:58.506836] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=8/60 SERVER_ID=19 SERVER="10.16.8.94:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 860 [2024-07-17 10:43:58.506840] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=9/60 SERVER_ID=8 SERVER="10.17.11.157:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 861 [2024-07-17 10:43:58.506845] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=10/60 SERVER_ID=21 SERVER="10.17.11.224:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 862 [2024-07-17 10:43:58.506849] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=11/60 SERVER_ID=35 SERVER="10.17.11.225:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 863 [2024-07-17 10:43:58.506853] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=12/60 SERVER_ID=17 SERVER="10.17.11.248:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 864 [2024-07-17 10:43:58.506858] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=13/60 SERVER_ID=25 SERVER="10.17.16.224:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 865 [2024-07-17 10:43:58.506862] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=14/60 SERVER_ID=24 SERVER="10.17.16.225:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 866 [2024-07-17 10:43:58.506866] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=15/60 SERVER_ID=76 SERVER="10.17.16.249:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 867 [2024-07-17 10:43:58.506870] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=16/60 SERVER_ID=38 SERVER="10.17.19.120:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 868 [2024-07-17 10:43:58.506874] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=17/60 SERVER_ID=9 SERVER="10.17.19.121:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 869 [2024-07-17 10:43:58.506878] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=18/60 SERVER_ID=26 SERVER="10.17.19.96:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 870 [2024-07-17 10:43:58.506886] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_SERVER_LIST] INDEX=19/60 SERVER_ID=34 SERVER="10.17.2.161:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 871 [2024-07-17 10:43:58.506892] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=20/60 SERVER_ID=74 SERVER="10.17.2.184:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 872 [2024-07-17 10:43:58.506898] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_SERVER_LIST] INDEX=21/60 SERVER_ID=69 SERVER="10.17.40.32:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 873 [2024-07-17 10:43:58.506902] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=22/60 SERVER_ID=16 SERVER="10.17.40.56:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 874 [2024-07-17 10:43:58.506906] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=23/60 SERVER_ID=1 SERVER="10.17.42.249:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 875 [2024-07-17 10:43:58.506911] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=24/60 SERVER_ID=18 SERVER="10.17.43.31:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 876 [2024-07-17 10:43:58.506915] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=25/60 SERVER_ID=83 SERVER="10.17.43.32:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 877 [2024-07-17 10:43:58.506919] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=26/60 SERVER_ID=3 SERVER="10.17.43.33:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 878 [2024-07-17 10:43:58.506923] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=27/60 SERVER_ID=11 SERVER="10.17.43.36:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 879 [2024-07-17 10:43:58.506927] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=28/60 SERVER_ID=78 SERVER="10.17.43.56:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 880 [2024-07-17 10:43:58.506932] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=29/60 SERVER_ID=65 SERVER="10.17.43.57:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 881 [2024-07-17 10:43:58.506936] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=30/60 SERVER_ID=68 SERVER="10.17.57.224:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 882 [2024-07-17 10:43:58.506940] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=31/60 SERVER_ID=37 SERVER="10.17.57.225:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 883 [2024-07-17 10:43:58.506944] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=32/60 SERVER_ID=72 SERVER="10.17.57.248:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 884 [2024-07-17 10:43:58.506948] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=33/60 SERVER_ID=5 SERVER="10.50.54.155:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 885 [2024-07-17 10:43:58.506952] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=34/60 SERVER_ID=29 SERVER="10.50.54.156:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 886 [2024-07-17 10:43:58.506956] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=35/60 SERVER_ID=28 SERVER="10.50.54.180:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 887 [2024-07-17 10:43:58.506962] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_SERVER_LIST] INDEX=36/60 SERVER_ID=81 SERVER="10.50.54.220:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 888 [2024-07-17 10:43:58.506966] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=37/60 SERVER_ID=22 SERVER="10.50.54.244:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 889 [2024-07-17 10:43:58.506970] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=38/60 SERVER_ID=30 SERVER="10.50.55.114:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 890 [2024-07-17 10:43:58.506974] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=39/60 SERVER_ID=13 SERVER="10.50.55.116:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 891 [2024-07-17 10:43:58.506978] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=40/60 SERVER_ID=14 SERVER="10.50.55.156:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 892 [2024-07-17 10:43:58.506982] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=41/60 SERVER_ID=10 SERVER="10.50.55.180:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 893 [2024-07-17 10:43:58.506986] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=42/60 SERVER_ID=7 SERVER="10.50.55.29:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 894 [2024-07-17 10:43:58.506990] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=43/60 SERVER_ID=2 SERVER="10.50.55.52:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 895 [2024-07-17 10:43:58.506994] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=44/60 SERVER_ID=71 SERVER="10.50.55.90:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 896 [2024-07-17 10:43:58.506998] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=45/60 SERVER_ID=15 SERVER="10.50.55.92:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 897 [2024-07-17 10:43:58.507002] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=46/60 SERVER_ID=27 SERVER="10.50.56.26:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 898 [2024-07-17 10:43:58.507006] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=47/60 SERVER_ID=12 SERVER="10.50.56.53:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 899 [2024-07-17 10:43:58.507010] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=48/60 SERVER_ID=85 SERVER="10.50.73.157:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 900 [2024-07-17 10:43:58.507015] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=49/60 SERVER_ID=20 SERVER="10.50.73.158:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 901 [2024-07-17 10:43:58.507019] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=50/60 SERVER_ID=67 SERVER="10.50.73.181:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 902 [2024-07-17 10:43:58.507024] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=51/60 SERVER_ID=77 SERVER="10.50.73.182:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 903 [2024-07-17 10:43:58.507028] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=52/60 SERVER_ID=32 SERVER="10.50.73.221:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 904 [2024-07-17 10:43:58.507033] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=53/60 SERVER_ID=79 SERVER="10.50.73.245:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 905 [2024-07-17 10:43:58.507036] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=54/60 SERVER_ID=6 SERVER="10.50.73.246:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 906 [2024-07-17 10:43:58.507040] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=55/60 SERVER_ID=80 SERVER="10.50.73.29:2882" STATUS=1(ACTIVE) ZONE=az4 REGION=BJ(LOW) VERSION=89 907 [2024-07-17 10:43:58.507044] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=56/60 SERVER_ID=66 SERVER="10.50.73.30:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 908 [2024-07-17 10:43:58.507049] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] INDEX=57/60 SERVER_ID=36 SERVER="10.50.73.54:2882" STATUS=1(ACTIVE) ZONE=az2 REGION=BJ(LOW) VERSION=89 909 [2024-07-17 10:43:58.507053] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=58/60 SERVER_ID=70 SERVER="10.50.73.93:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 910 [2024-07-17 10:43:58.507057] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] INDEX=59/60 SERVER_ID=23 SERVER="10.50.75.219:2882" STATUS=1(ACTIVE) ZONE=az1 REGION=BJ(LOW) VERSION=89 911 [2024-07-17 10:43:58.507060] INFO [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:525) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] COUNT=60 VERSION=89 912 [2024-07-17 10:43:58.507077] INFO [TLOG] purge_stale_records_ (ob_log_all_svr_cache.cpp:646) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] [PURGE] PURGE_COUNT=0 CUR_COUNT=60 VERSION=89 913 [2024-07-17 10:43:58.512401] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501148-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 914 [2024-07-17 10:43:58.512436] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501148-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 915 [2024-07-17 10:43:58.533513] INFO [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] add_server_or_update(next_svr_index=0, svr_items=[{server:"10.16.52.121:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}]) 916 [2024-07-17 10:43:58.533534] INFO [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=20] add_server_or_update(next_svr_index=0, svr_items=[{server:"10.16.52.121:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"10.16.8.94:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}]) 917 [2024-07-17 10:43:58.533541] INFO [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=7] add_server_or_update(next_svr_index=0, svr_items=[{server:"10.16.52.121:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"10.16.8.94:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}, {server:"10.17.16.225:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}]) 918 [2024-07-17 10:43:58.533561] INFO [TLOG] get_ls_svr_list_ (ob_log_route_service.cpp:817) [45031][LogRouter][T0][Y0-0000000000000000-0-0] [lt=16] get_ls_svr_list_ done(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}, svr_list={next_svr_index:0, svr_num:3, svr_items:[{server:"10.16.52.121:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"10.17.16.225:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"10.16.8.94:2882", ranges:[LSN:{94282199040, 94453866374}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}]}, tg_id_=259) 919 [2024-07-17 10:43:58.561381] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 920 [2024-07-17 10:43:58.561401] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501147-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A4-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501148-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 921 [2024-07-17 10:43:58.561427] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 922 [2024-07-17 10:43:58.561438] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=10][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501147-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A4-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501148-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 923 [2024-07-17 10:43:58.623398] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45019][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 924 [2024-07-17 10:43:58.623468] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45021][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 925 [2024-07-17 10:43:58.623733] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45018][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 926 [2024-07-17 10:43:58.624097] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45020][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 927 [2024-07-17 10:43:58.661525] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 928 [2024-07-17 10:43:58.661565] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=27] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 929 [2024-07-17 10:43:58.679049] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501149-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 930 [2024-07-17 10:43:58.679088] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501149-0-0] [lt=37] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 931 [2024-07-17 10:43:58.695679] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A5-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 932 [2024-07-17 10:43:58.695715] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A5-0-0] [lt=34] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 933 [2024-07-17 10:43:58.713110] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050114A-0-0] [lt=21] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 934 [2024-07-17 10:43:58.713141] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050114A-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 935 [2024-07-17 10:43:58.761651] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 936 [2024-07-17 10:43:58.761677] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501149-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A5-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114A-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 937 [2024-07-17 10:43:58.761712] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 938 [2024-07-17 10:43:58.761717] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501149-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A5-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 939 [2024-07-17 10:43:58.844850] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45025][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 940 [2024-07-17 10:43:58.845139] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45027][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 941 [2024-07-17 10:43:58.845350] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45023][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 942 [2024-07-17 10:43:58.845666] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45029][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 943 [2024-07-17 10:43:58.861812] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 944 [2024-07-17 10:43:58.861850] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 945 [2024-07-17 10:43:58.879635] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050114B-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 946 [2024-07-17 10:43:58.879673] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050114B-0-0] [lt=36] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 947 [2024-07-17 10:43:58.895230] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A6-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 948 [2024-07-17 10:43:58.895257] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A6-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 949 [2024-07-17 10:43:58.912691] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050114C-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 950 [2024-07-17 10:43:58.912719] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050114C-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 951 [2024-07-17 10:43:58.932696] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45036][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=3/0, request done=1660/0, request doing=0/0) 952 [2024-07-17 10:43:58.932739] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45034][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=3/0, request done=1661/0, request doing=0/0) 953 [2024-07-17 10:43:58.933281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45035][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=3/0, request done=1660/0, request doing=0/0) 954 [2024-07-17 10:43:58.933282] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45033][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=3/0, request done=1661/0, request doing=0/0) 955 [2024-07-17 10:43:58.933299] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45042][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 956 [2024-07-17 10:43:58.934606] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [45038][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 957 [2024-07-17 10:43:58.934607] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [45040][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 958 [2024-07-17 10:43:58.934788] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [45044][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) 959 [2024-07-17 10:43:58.961935] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 960 [2024-07-17 10:43:58.961953] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A6-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114C-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 961 [2024-07-17 10:43:58.961980] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 962 [2024-07-17 10:43:58.961985] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114B-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A6-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 963 [2024-07-17 10:43:58.962265] INFO eloop_run (eloop.c:140) [45010][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1721184238962264, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 964 [2024-07-17 10:43:58.970709] INFO eloop_run (eloop.c:140) [45014][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1721184238970708, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 965 [2024-07-17 10:43:59.062061] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 966 [2024-07-17 10:43:59.062093] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=21] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 967 [2024-07-17 10:43:59.079419] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050114D-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 968 [2024-07-17 10:43:59.079446] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050114D-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 969 [2024-07-17 10:43:59.096006] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A7-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 970 [2024-07-17 10:43:59.096033] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A7-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 971 [2024-07-17 10:43:59.113178] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050114E-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 972 [2024-07-17 10:43:59.113200] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050114E-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 973 [2024-07-17 10:43:59.162140] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 974 [2024-07-17 10:43:59.162159] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A7-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114E-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 975 [2024-07-17 10:43:59.162186] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=26][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 976 [2024-07-17 10:43:59.162191] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114D-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A7-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 977 [2024-07-17 10:43:59.262275] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=23] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 978 [2024-07-17 10:43:59.262310] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=25] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 979 [2024-07-17 10:43:59.279704] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-000000000050114F-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 980 [2024-07-17 10:43:59.279731] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-000000000050114F-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 981 [2024-07-17 10:43:59.295716] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A8-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 982 [2024-07-17 10:43:59.295741] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A8-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 983 [2024-07-17 10:43:59.313123] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501150-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 984 [2024-07-17 10:43:59.313148] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501150-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 985 [2024-07-17 10:43:59.362386] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=16][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 986 [2024-07-17 10:43:59.362399] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=13][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A8-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501150-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 987 [2024-07-17 10:43:59.362422] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 988 [2024-07-17 10:43:59.362427] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-000000000050114F-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A8-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501150-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 989 [2024-07-17 10:43:59.462519] INFO [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=17] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:128, tenant_id:1004, ls_id:{id:1}}) 990 [2024-07-17 10:43:59.462552] INFO [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:883) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=18] start lsn locate request launched(tls_id={tenant_id:1004, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2024-07-16 15:14:45.070225]", svr_cnt=3, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.16.52.121:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"10.17.16.225:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}]) 991 [2024-07-17 10:43:59.479944] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501151-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.8.94:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 992 [2024-07-17 10:43:59.479969] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501151-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.8.94:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3) 993 [2024-07-17 10:43:59.496040] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45047][][T0][YAED10A5818EC-00000000006008A9-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.16.52.121:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 994 [2024-07-17 10:43:59.496064] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45047][][T0][YAED10A5818EC-00000000006008A9-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.16.52.121:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3) 995 [2024-07-17 10:43:59.513394] INFO [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [45049][][T0][YAED10A5818EC-0000000000501152-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1004, svr="10.17.16.225:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1721114085070225350}], client_id:{client_addr:"10.88.24.236:44753", client_pid:44753}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]}) 996 [2024-07-17 10:43:59.513415] INFO [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:895) [45049][][T0][YAED10A5818EC-0000000000501152-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1004, ls_id:{id:1}}, svr_identity="10.17.16.225:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1721114085070225350, delta=-1721114085070225351, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3) 997 [2024-07-17 10:43:59.562622] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=14][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=929, info="request start lsn from all server fail") 998 [2024-07-17 10:43:59.562633] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:929) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=10][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns=1721114085070225350, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501151-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A9-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501152-0-0}]) BACKTRACE:0x1e274c4d 0xe3fc8cf 0xe3fc461 0xe3fc0ca 0xe3fbebb 0xe6d04c0 0xe516530 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d 999 [2024-07-17 10:43:59.562652] ERROR issue_dba_error (ob_log.cpp:1891) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=908, info="start lsn locate fail") 1000 [2024-07-17 10:43:59.562657] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [45052][][T0][YAED10A5818EC-0000000000200001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1004, ls_id:{id:1}}, start_tstamp_ns:1721114085070225350, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"10.16.8.94:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501151-0-0}, {svr:"10.16.52.121:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-00000000006008A9-0-0}, {svr:"10.17.16.225:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:YAED10A5818EC-0000000000501152-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1e274c4d 0xbd28dc0 0xe42a76c 0xe42a572 0xbd570f5 0xe517fdf 0xe516545 0xe4a263a 0xe4a1a23 0xe4a0e78 0xe57bbcf 0x7f273e09bea5 0x7f273d6b5b0d