1 [2025-08-28 16:25:52.944801] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F39-0-0] [lt=44] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
2 [2025-08-28 16:25:52.946746] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F39-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
3 [2025-08-28 16:25:52.946764] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F39-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
4 [2025-08-28 16:25:52.949082] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F39-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
5 [2025-08-28 16:25:52.949098] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F39-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
6 [2025-08-28 16:25:53.042655] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
7 [2025-08-28 16:25:53.042713] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=57][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F39-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F39-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F39-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
8 [2025-08-28 16:25:53.042774] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=60][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
9 [2025-08-28 16:25:53.042780] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F39-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F39-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F39-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
10 [2025-08-28 16:25:53.142860] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=18] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
11 [2025-08-28 16:25:53.142927] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=49] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
12 [2025-08-28 16:25:53.145002] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F3A-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
13 [2025-08-28 16:25:53.145028] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F3A-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
14 [2025-08-28 16:25:53.147053] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F3A-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
15 [2025-08-28 16:25:53.147072] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F3A-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
16 [2025-08-28 16:25:53.149710] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F3A-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
17 [2025-08-28 16:25:53.149733] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F3A-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
18 [2025-08-28 16:25:53.217862] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
19 [2025-08-28 16:25:53.217958] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
20 [2025-08-28 16:25:53.218171] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
21 [2025-08-28 16:25:53.218171] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
22 [2025-08-28 16:25:53.243017] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
23 [2025-08-28 16:25:53.243039] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3A-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3A-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3A-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
24 [2025-08-28 16:25:53.243081] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
25 [2025-08-28 16:25:53.243088] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=6][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3A-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3A-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
26 [2025-08-28 16:25:53.243245] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1756369553243244, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0
27 [2025-08-28 16:25:53.289290] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1756369553289287, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0
28 [2025-08-28 16:25:53.343177] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
29 [2025-08-28 16:25:53.343233] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
30 [2025-08-28 16:25:53.345319] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F3B-0-0] [lt=21] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
31 [2025-08-28 16:25:53.345348] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F3B-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
32 [2025-08-28 16:25:53.347689] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F3B-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
33 [2025-08-28 16:25:53.347709] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F3B-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
34 [2025-08-28 16:25:53.350132] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F3B-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
35 [2025-08-28 16:25:53.350149] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F3B-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
36 [2025-08-28 16:25:53.437466] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=34] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
37 [2025-08-28 16:25:53.438458] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
38 [2025-08-28 16:25:53.438470] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
39 [2025-08-28 16:25:53.438493] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
40 [2025-08-28 16:25:53.443312] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
41 [2025-08-28 16:25:53.443330] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=17][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3B-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3B-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3B-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
42 [2025-08-28 16:25:53.443356] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
43 [2025-08-28 16:25:53.443360] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=3][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3B-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3B-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3B-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
44 [2025-08-28 16:25:53.466801] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=3/0, request done=5996/0, request doing=0/0)
45 [2025-08-28 16:25:53.466891] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=3/0, request done=5996/0, request doing=0/0)
46 [2025-08-28 16:25:53.467250] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=5997/0, request doing=0/0)
47 [2025-08-28 16:25:53.467413] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=5996/0, request doing=0/0)
48 [2025-08-28 16:25:53.468413] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
49 [2025-08-28 16:25:53.468461] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
50 [2025-08-28 16:25:53.468462] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
51 [2025-08-28 16:25:53.469464] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
52 [2025-08-28 16:25:53.543463] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=20] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
53 [2025-08-28 16:25:53.543519] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
54 [2025-08-28 16:25:53.545636] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F3C-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
55 [2025-08-28 16:25:53.545663] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F3C-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
56 [2025-08-28 16:25:53.547640] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F3C-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
57 [2025-08-28 16:25:53.547670] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F3C-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
58 [2025-08-28 16:25:53.550109] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F3C-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
59 [2025-08-28 16:25:53.550126] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F3C-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
60 [2025-08-28 16:25:53.643608] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
61 [2025-08-28 16:25:53.643644] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3C-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3C-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3C-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
62 [2025-08-28 16:25:53.643673] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
63 [2025-08-28 16:25:53.643677] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3C-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3C-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
64 [2025-08-28 16:25:53.743734] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
65 [2025-08-28 16:25:53.743794] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=39] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
66 [2025-08-28 16:25:53.746046] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F3D-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
67 [2025-08-28 16:25:53.746076] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F3D-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
68 [2025-08-28 16:25:53.748007] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F3D-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
69 [2025-08-28 16:25:53.748039] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F3D-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
70 [2025-08-28 16:25:53.750482] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F3D-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
71 [2025-08-28 16:25:53.750516] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F3D-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
72 [2025-08-28 16:25:53.843860] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
73 [2025-08-28 16:25:53.843895] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3D-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3D-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3D-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
74 [2025-08-28 16:25:53.843927] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
75 [2025-08-28 16:25:53.843932] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3D-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3D-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3D-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
76 [2025-08-28 16:25:53.944036] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
77 [2025-08-28 16:25:53.944086] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
78 [2025-08-28 16:25:53.946074] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F3E-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
79 [2025-08-28 16:25:53.946102] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F3E-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
80 [2025-08-28 16:25:53.948042] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F3E-0-0] [lt=19] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
81 [2025-08-28 16:25:53.948067] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F3E-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
82 [2025-08-28 16:25:53.950703] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F3E-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
83 [2025-08-28 16:25:53.950727] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F3E-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
84 [2025-08-28 16:25:54.044178] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
85 [2025-08-28 16:25:54.044212] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=33][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3E-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3E-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3E-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
86 [2025-08-28 16:25:54.044250] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
87 [2025-08-28 16:25:54.044256] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3E-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3E-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
88 [2025-08-28 16:25:54.144373] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
89 [2025-08-28 16:25:54.144431] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
90 [2025-08-28 16:25:54.146491] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F3F-0-0] [lt=23] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
91 [2025-08-28 16:25:54.146523] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F3F-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
92 [2025-08-28 16:25:54.148549] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F3F-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
93 [2025-08-28 16:25:54.148583] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F3F-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
94 [2025-08-28 16:25:54.151129] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F3F-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
95 [2025-08-28 16:25:54.151159] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F3F-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
96 [2025-08-28 16:25:54.217867] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
97 [2025-08-28 16:25:54.218102] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
98 [2025-08-28 16:25:54.218463] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
99 [2025-08-28 16:25:54.218975] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
100 [2025-08-28 16:25:54.244544] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
101 [2025-08-28 16:25:54.244577] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3F-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3F-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3F-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
102 [2025-08-28 16:25:54.244616] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=38][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
103 [2025-08-28 16:25:54.244622] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=6][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F3F-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F3F-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F3F-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
104 [2025-08-28 16:25:54.250864] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1756369554250861, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0
105 [2025-08-28 16:25:54.296907] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1756369554296902, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0
106 [2025-08-28 16:25:54.344725] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
107 [2025-08-28 16:25:54.344792] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=46] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
108 [2025-08-28 16:25:54.346855] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F40-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
109 [2025-08-28 16:25:54.346882] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F40-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
110 [2025-08-28 16:25:54.349402] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F40-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
111 [2025-08-28 16:25:54.349432] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F40-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
112 [2025-08-28 16:25:54.351859] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F40-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
113 [2025-08-28 16:25:54.351877] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F40-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
114 [2025-08-28 16:25:54.438468] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
115 [2025-08-28 16:25:54.438465] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
116 [2025-08-28 16:25:54.438493] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
117 [2025-08-28 16:25:54.438507] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
118 [2025-08-28 16:25:54.444857] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
119 [2025-08-28 16:25:54.444872] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=14][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F40-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F40-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F40-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
120 [2025-08-28 16:25:54.444904] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
121 [2025-08-28 16:25:54.444909] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F40-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F40-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F40-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
122 [2025-08-28 16:25:54.466984] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=3/0, request done=6000/0, request doing=0/0)
123 [2025-08-28 16:25:54.466984] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=3/0, request done=6000/0, request doing=0/0)
124 [2025-08-28 16:25:54.467254] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=6000/0, request doing=0/0)
125 [2025-08-28 16:25:54.467513] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6000/0, request doing=0/0)
126 [2025-08-28 16:25:54.467965] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
127 [2025-08-28 16:25:54.467970] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
128 [2025-08-28 16:25:54.468454] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
129 [2025-08-28 16:25:54.468851] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
130 [2025-08-28 16:25:54.544962] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=14] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
131 [2025-08-28 16:25:54.545019] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
132 [2025-08-28 16:25:54.547627] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F41-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
133 [2025-08-28 16:25:54.547651] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F41-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
134 [2025-08-28 16:25:54.549705] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F41-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
135 [2025-08-28 16:25:54.549723] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F41-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
136 [2025-08-28 16:25:54.552231] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F41-0-0] [lt=7] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
137 [2025-08-28 16:25:54.552249] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F41-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
138 [2025-08-28 16:25:54.645300] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
139 [2025-08-28 16:25:54.645334] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=33][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F41-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F41-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F41-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
140 [2025-08-28 16:25:54.645371] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
141 [2025-08-28 16:25:54.645377] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F41-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F41-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F41-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
142 [2025-08-28 16:25:54.745427] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
143 [2025-08-28 16:25:54.745494] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=47] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
144 [2025-08-28 16:25:54.747820] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F42-0-0] [lt=26] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
145 [2025-08-28 16:25:54.747847] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F42-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
146 [2025-08-28 16:25:54.749868] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F42-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
147 [2025-08-28 16:25:54.749890] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F42-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
148 [2025-08-28 16:25:54.752540] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F42-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
149 [2025-08-28 16:25:54.752558] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F42-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
150 [2025-08-28 16:25:54.845605] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
151 [2025-08-28 16:25:54.845653] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=47][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F42-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F42-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F42-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
152 [2025-08-28 16:25:54.845698] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=44][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
153 [2025-08-28 16:25:54.845702] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F42-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F42-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F42-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
154 [2025-08-28 16:25:54.945787] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
155 [2025-08-28 16:25:54.945854] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=39] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
156 [2025-08-28 16:25:54.948149] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F43-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
157 [2025-08-28 16:25:54.948188] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F43-0-0] [lt=36] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
158 [2025-08-28 16:25:54.950184] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F43-0-0] [lt=20] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
159 [2025-08-28 16:25:54.950203] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F43-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
160 [2025-08-28 16:25:54.952683] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F43-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
161 [2025-08-28 16:25:54.952701] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F43-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
162 [2025-08-28 16:25:55.045950] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
163 [2025-08-28 16:25:55.045987] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F43-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F43-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F43-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
164 [2025-08-28 16:25:55.046028] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=40][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
165 [2025-08-28 16:25:55.046033] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F43-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F43-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F43-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
166 [2025-08-28 16:25:55.146122] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
167 [2025-08-28 16:25:55.146169] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
168 [2025-08-28 16:25:55.148238] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F44-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
169 [2025-08-28 16:25:55.148275] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F44-0-0] [lt=34] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
170 [2025-08-28 16:25:55.150198] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F44-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
171 [2025-08-28 16:25:55.150222] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F44-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
172 [2025-08-28 16:25:55.152723] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F44-0-0] [lt=7] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
173 [2025-08-28 16:25:55.152742] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F44-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
174 [2025-08-28 16:25:55.218334] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
175 [2025-08-28 16:25:55.218881] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
176 [2025-08-28 16:25:55.218882] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
177 [2025-08-28 16:25:55.218906] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
178 [2025-08-28 16:25:55.246264] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
179 [2025-08-28 16:25:55.246302] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F44-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F44-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F44-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
180 [2025-08-28 16:25:55.246332] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
181 [2025-08-28 16:25:55.246337] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F44-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F44-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F44-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
182 [2025-08-28 16:25:55.258477] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1756369555258474, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0
183 [2025-08-28 16:25:55.304519] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1756369555304516, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0
184 [2025-08-28 16:25:55.346425] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
185 [2025-08-28 16:25:55.346492] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=46] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
186 [2025-08-28 16:25:55.348647] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F45-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
187 [2025-08-28 16:25:55.348678] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F45-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
188 [2025-08-28 16:25:55.350628] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F45-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
189 [2025-08-28 16:25:55.350651] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F45-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
190 [2025-08-28 16:25:55.353096] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F45-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
191 [2025-08-28 16:25:55.353118] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F45-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
192 [2025-08-28 16:25:55.438468] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
193 [2025-08-28 16:25:55.438467] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
194 [2025-08-28 16:25:55.438518] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
195 [2025-08-28 16:25:55.438535] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
196 [2025-08-28 16:25:55.446598] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
197 [2025-08-28 16:25:55.446618] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=19][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F45-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F45-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F45-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
198 [2025-08-28 16:25:55.446649] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
199 [2025-08-28 16:25:55.446653] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F45-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F45-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F45-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
200 [2025-08-28 16:25:55.466752] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=3/0, request done=6004/0, request doing=0/0)
201 [2025-08-28 16:25:55.466753] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=3/0, request done=6004/0, request doing=0/0)
202 [2025-08-28 16:25:55.466961] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=3/0, request done=6003/0, request doing=0/0)
203 [2025-08-28 16:25:55.467213] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6004/0, request doing=0/0)
204 [2025-08-28 16:25:55.468460] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
205 [2025-08-28 16:25:55.468472] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
206 [2025-08-28 16:25:55.468476] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
207 [2025-08-28 16:25:55.468851] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
208 [2025-08-28 16:25:55.546742] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
209 [2025-08-28 16:25:55.546799] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
210 [2025-08-28 16:25:55.549166] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F46-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
211 [2025-08-28 16:25:55.549189] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F46-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
212 [2025-08-28 16:25:55.551743] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F46-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
213 [2025-08-28 16:25:55.551770] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F46-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
214 [2025-08-28 16:25:55.554486] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F46-0-0] [lt=28] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
215 [2025-08-28 16:25:55.554513] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F46-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
216 [2025-08-28 16:25:55.646894] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
217 [2025-08-28 16:25:55.646931] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F46-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F46-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F46-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
218 [2025-08-28 16:25:55.646961] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
219 [2025-08-28 16:25:55.646965] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F46-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F46-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F46-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
220 [2025-08-28 16:25:55.747065] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=26] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
221 [2025-08-28 16:25:55.747122] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
222 [2025-08-28 16:25:55.749172] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F47-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
223 [2025-08-28 16:25:55.749199] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F47-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
224 [2025-08-28 16:25:55.751318] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F47-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
225 [2025-08-28 16:25:55.751345] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F47-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
226 [2025-08-28 16:25:55.753722] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F47-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
227 [2025-08-28 16:25:55.753740] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F47-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
228 [2025-08-28 16:25:55.847222] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
229 [2025-08-28 16:25:55.847247] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F47-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F47-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F47-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
230 [2025-08-28 16:25:55.847279] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
231 [2025-08-28 16:25:55.847294] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F47-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F47-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F47-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
232 [2025-08-28 16:25:55.947389] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
233 [2025-08-28 16:25:55.947461] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=42] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
234 [2025-08-28 16:25:55.949848] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F48-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
235 [2025-08-28 16:25:55.949879] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F48-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
236 [2025-08-28 16:25:55.951813] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F48-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
237 [2025-08-28 16:25:55.951834] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F48-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
238 [2025-08-28 16:25:55.954253] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F48-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
239 [2025-08-28 16:25:55.954282] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F48-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
240 [2025-08-28 16:25:56.047554] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
241 [2025-08-28 16:25:56.047590] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F48-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F48-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F48-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
242 [2025-08-28 16:25:56.047637] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=46][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
243 [2025-08-28 16:25:56.047641] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F48-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F48-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F48-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
244 [2025-08-28 16:25:56.147690] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
245 [2025-08-28 16:25:56.147733] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
246 [2025-08-28 16:25:56.150931] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F49-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
247 [2025-08-28 16:25:56.150955] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F49-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
248 [2025-08-28 16:25:56.152970] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F49-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
249 [2025-08-28 16:25:56.152989] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F49-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
250 [2025-08-28 16:25:56.155545] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F49-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
251 [2025-08-28 16:25:56.155564] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F49-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
252 [2025-08-28 16:25:56.218089] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
253 [2025-08-28 16:25:56.218523] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
254 [2025-08-28 16:25:56.218853] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
255 [2025-08-28 16:25:56.218853] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
256 [2025-08-28 16:25:56.247830] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
257 [2025-08-28 16:25:56.247854] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F49-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F49-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F49-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
258 [2025-08-28 16:25:56.247896] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
259 [2025-08-28 16:25:56.247902] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=6][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F49-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F49-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F49-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
260 [2025-08-28 16:25:56.266095] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1756369556266092, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0
261 [2025-08-28 16:25:56.312198] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1756369556312195, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007679, add_bytes: 0
262 [2025-08-28 16:25:56.347999] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=26] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
263 [2025-08-28 16:25:56.348069] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
264 [2025-08-28 16:25:56.350287] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F4A-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
265 [2025-08-28 16:25:56.350315] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F4A-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
266 [2025-08-28 16:25:56.352275] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F4A-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
267 [2025-08-28 16:25:56.352303] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F4A-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
268 [2025-08-28 16:25:56.355035] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F4A-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
269 [2025-08-28 16:25:56.355053] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F4A-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
270 [2025-08-28 16:25:56.437863] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=34] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
271 [2025-08-28 16:25:56.437863] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
272 [2025-08-28 16:25:56.437920] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
273 [2025-08-28 16:25:56.438465] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
274 [2025-08-28 16:25:56.448166] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
275 [2025-08-28 16:25:56.448189] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4A-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4A-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4A-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
276 [2025-08-28 16:25:56.448228] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=38][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
277 [2025-08-28 16:25:56.448233] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4A-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4A-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
278 [2025-08-28 16:25:56.467071] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=3/0, request done=6008/0, request doing=0/0)
279 [2025-08-28 16:25:56.467313] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6007/0, request doing=0/0)
280 [2025-08-28 16:25:56.467398] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=3/0, request done=6008/0, request doing=0/0)
281 [2025-08-28 16:25:56.467572] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=3/0, request done=6007/0, request doing=0/0)
282 [2025-08-28 16:25:56.468458] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
283 [2025-08-28 16:25:56.468459] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
284 [2025-08-28 16:25:56.468570] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
285 [2025-08-28 16:25:56.468851] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
286 [2025-08-28 16:25:56.548320] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
287 [2025-08-28 16:25:56.548398] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=40] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
288 [2025-08-28 16:25:56.550525] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F4B-0-0] [lt=24] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
289 [2025-08-28 16:25:56.550552] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F4B-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
290 [2025-08-28 16:25:56.552603] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F4B-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
291 [2025-08-28 16:25:56.552636] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F4B-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
292 [2025-08-28 16:25:56.555175] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F4B-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
293 [2025-08-28 16:25:56.555193] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F4B-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
294 [2025-08-28 16:25:56.648473] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
295 [2025-08-28 16:25:56.648510] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4B-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4B-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4B-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
296 [2025-08-28 16:25:56.648547] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
297 [2025-08-28 16:25:56.648552] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4B-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4B-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4B-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
298 [2025-08-28 16:25:56.748650] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
299 [2025-08-28 16:25:56.748697] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=27] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
300 [2025-08-28 16:25:56.750823] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F4C-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
301 [2025-08-28 16:25:56.750847] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F4C-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
302 [2025-08-28 16:25:56.752862] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F4C-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
303 [2025-08-28 16:25:56.752882] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F4C-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
304 [2025-08-28 16:25:56.755334] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F4C-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
305 [2025-08-28 16:25:56.755362] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F4C-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
306 [2025-08-28 16:25:56.848800] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
307 [2025-08-28 16:25:56.848828] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=27][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4C-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4C-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4C-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
308 [2025-08-28 16:25:56.848872] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=43][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
309 [2025-08-28 16:25:56.848876] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4C-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4C-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
310 [2025-08-28 16:25:56.948967] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=19] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
311 [2025-08-28 16:25:56.949033] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=45] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
312 [2025-08-28 16:25:56.951101] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F4D-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
313 [2025-08-28 16:25:56.951131] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F4D-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
314 [2025-08-28 16:25:56.953069] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F4D-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
315 [2025-08-28 16:25:56.953091] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F4D-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
316 [2025-08-28 16:25:56.955632] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F4D-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
317 [2025-08-28 16:25:56.955648] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F4D-0-0] [lt=15] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
318 [2025-08-28 16:25:57.049094] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
319 [2025-08-28 16:25:57.049151] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=56][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4D-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4D-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4D-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
320 [2025-08-28 16:25:57.049182] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
321 [2025-08-28 16:25:57.049186] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4D-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4D-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4D-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
322 [2025-08-28 16:25:57.149271] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
323 [2025-08-28 16:25:57.149319] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
324 [2025-08-28 16:25:57.151678] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F4E-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
325 [2025-08-28 16:25:57.151717] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F4E-0-0] [lt=36] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
326 [2025-08-28 16:25:57.153696] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F4E-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
327 [2025-08-28 16:25:57.153723] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F4E-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
328 [2025-08-28 16:25:57.156473] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F4E-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
329 [2025-08-28 16:25:57.156488] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F4E-0-0] [lt=14] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
330 [2025-08-28 16:25:57.217474] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
331 [2025-08-28 16:25:57.218329] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
332 [2025-08-28 16:25:57.218409] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
333 [2025-08-28 16:25:57.218851] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
334 [2025-08-28 16:25:57.249406] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
335 [2025-08-28 16:25:57.249433] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=26][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4E-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4E-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4E-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
336 [2025-08-28 16:25:57.249475] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
337 [2025-08-28 16:25:57.249480] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4E-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4E-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
338 [2025-08-28 16:25:57.273709] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1756369557273707, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0
339 [2025-08-28 16:25:57.319830] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1756369557319827, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007632, add_bytes: 0
340 [2025-08-28 16:25:57.349569] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=14] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
341 [2025-08-28 16:25:57.349612] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
342 [2025-08-28 16:25:57.351717] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F4F-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
343 [2025-08-28 16:25:57.351755] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F4F-0-0] [lt=35] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
344 [2025-08-28 16:25:57.353895] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F4F-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
345 [2025-08-28 16:25:57.353944] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F4F-0-0] [lt=46] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
346 [2025-08-28 16:25:57.357194] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F4F-0-0] [lt=7] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
347 [2025-08-28 16:25:57.357224] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F4F-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
348 [2025-08-28 16:25:57.437856] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
349 [2025-08-28 16:25:57.438473] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
350 [2025-08-28 16:25:57.438496] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
351 [2025-08-28 16:25:57.438851] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
352 [2025-08-28 16:25:57.449730] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
353 [2025-08-28 16:25:57.449759] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4F-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4F-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4F-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
354 [2025-08-28 16:25:57.449791] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
355 [2025-08-28 16:25:57.449796] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F4F-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F4F-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F4F-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
356 [2025-08-28 16:25:57.466839] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6011/0, request doing=0/0)
357 [2025-08-28 16:25:57.467008] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6011/0, request doing=0/0)
358 [2025-08-28 16:25:57.467219] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6012/0, request doing=0/0)
359 [2025-08-28 16:25:57.467747] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=3/0, request done=6011/0, request doing=0/0)
360 [2025-08-28 16:25:57.468467] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
361 [2025-08-28 16:25:57.468478] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
362 [2025-08-28 16:25:57.468512] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
363 [2025-08-28 16:25:57.468854] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
364 [2025-08-28 16:25:57.549894] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
365 [2025-08-28 16:25:57.549957] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=46] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
366 [2025-08-28 16:25:57.553640] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F50-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
367 [2025-08-28 16:25:57.553676] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F50-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
368 [2025-08-28 16:25:57.556408] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F50-0-0] [lt=26] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
369 [2025-08-28 16:25:57.556435] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F50-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
370 [2025-08-28 16:25:57.560895] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F50-0-0] [lt=19] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
371 [2025-08-28 16:25:57.560921] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F50-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
372 [2025-08-28 16:25:57.577353] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=13] [STAT] [ALL_ZONE] INDEX=0/3 ZONE=zone1 REGION=sys_region STORAGE_TYPE=LOCAL VERSION=321
373 [2025-08-28 16:25:57.577380] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=24] [STAT] [ALL_ZONE] INDEX=1/3 ZONE=zone2 REGION=sys_region STORAGE_TYPE=LOCAL VERSION=321
374 [2025-08-28 16:25:57.577386] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_ZONE] INDEX=2/3 ZONE=zone3 REGION=sys_region STORAGE_TYPE=LOCAL VERSION=321
375 [2025-08-28 16:25:57.577391] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] INDEX=0/3 ZONE=zone1 ZONE_TYPE=ReadWrite VERSION=321
376 [2025-08-28 16:25:57.577396] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_ZONE] INDEX=1/3 ZONE=zone2 ZONE_TYPE=ReadWrite VERSION=321
377 [2025-08-28 16:25:57.577400] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] INDEX=2/3 ZONE=zone3 ZONE_TYPE=ReadWrite VERSION=321
378 [2025-08-28 16:25:57.577403] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:454) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] COUNT=3 VERSION=321
379 [2025-08-28 16:25:57.577418] INFO  [TLOG] purge_stale_zone_records_ (ob_log_all_svr_cache.cpp:660) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=9] [STAT] [ALL_ZONE] [PURGE] PURGE_COUNT=0 CUR_COUNT=3 VERSION=321
380 [2025-08-28 16:25:57.582492] INFO  [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [ALL_SERVER_LIST] INDEX=0/3 SERVER_ID=2 SERVER="172.17.0.11:2882" STATUS=1(ACTIVE) ZONE=zone2 REGION=sys_region(LOW) VERSION=321
381 [2025-08-28 16:25:57.582517] INFO  [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=23] [STAT] [ALL_SERVER_LIST] INDEX=1/3 SERVER_ID=1 SERVER="172.17.0.3:2882" STATUS=1(ACTIVE) ZONE=zone1 REGION=sys_region(LOW) VERSION=321
382 [2025-08-28 16:25:57.582523] INFO  [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=2/3 SERVER_ID=3 SERVER="172.17.16.5:2882" STATUS=1(ACTIVE) ZONE=zone3 REGION=sys_region(LOW) VERSION=321
383 [2025-08-28 16:25:57.582527] INFO  [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:525) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] COUNT=3 VERSION=321
384 [2025-08-28 16:25:57.582535] INFO  [TLOG] purge_stale_records_ (ob_log_all_svr_cache.cpp:646) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] [PURGE] PURGE_COUNT=0 CUR_COUNT=3 VERSION=321
385 [2025-08-28 16:25:57.589085] INFO  [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] add_server_or_update(next_svr_index=0, svr_items=[{server:"172.17.0.11:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}])
386 [2025-08-28 16:25:57.589103] INFO  [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=17] add_server_or_update(next_svr_index=0, svr_items=[{server:"172.17.0.11:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"172.17.0.3:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}])
387 [2025-08-28 16:25:57.589110] INFO  [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=7] add_server_or_update(next_svr_index=0, svr_items=[{server:"172.17.0.11:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"172.17.0.3:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}, {server:"172.17.16.5:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}])
388 [2025-08-28 16:25:57.589121] INFO  [TLOG] get_ls_svr_list_ (ob_log_route_service.cpp:817) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=8] get_ls_svr_list_ done(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}}, svr_list={next_svr_index:0, svr_num:3, svr_items:[{server:"172.17.0.11:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"172.17.16.5:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"172.17.0.3:2882", ranges:[LSN:{56636424192, 56718789377}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}]}, tg_id_=259)
389 [2025-08-28 16:25:57.619212] INFO  [TLOG.FETCHER] print_stat (ob_log_fetch_stream_container_mgr.cpp:178) [11788][][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [FS_CONTAINER_POOL] USED=1 FREE=0 FIXED=16 DYNAMIC=0
390 [2025-08-28 16:25:57.619234] INFO  [TLOG.FETCHER] print_stat (ob_log_fetch_stream_pool.cpp:76) [11788][][T0][Y0-0000000000000000-0-0] [lt=21] [STAT] [FETCH_STREAM_POOL] USED=0 FREE=0 FIXED=16 DYNAMIC=0
391 [2025-08-28 16:25:57.619247] INFO  [TLOG.FETCHER] print_stat (ob_log_fetch_log_rpc.cpp:1400) [11788][][T0][Y0-0000000000000000-0-0] [lt=13] [STAT] [RPC_RESULT_POOL] USED=0 FREE=0 FIXED=16 DYNAMIC=0
392 [2025-08-28 16:25:57.619273] INFO  [TLOG.FETCHER] print_fetcher_stat_ (ob_log_fetcher.cpp:840) [11788][][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] [FETCHER](upper_limit="[2025-08-27 20:22:26.397904]", dml_progress_limit_sec=30, fetcher_delay="[20:04:01.221360]")
393 [2025-08-28 16:25:57.619306] INFO  [TLOG.FETCHER] print_k_slowest_ls (ob_log_ls_fetch_mgr.cpp:429) [11788][][T0][Y0-0000000000000000-0-0] [lt=8] [STAT] slow fetch progress start. ls_count=1/1
394 [2025-08-28 16:25:57.619321] INFO  [TLOG.FETCHER] print_fetch_progress (ob_log_ls_fetch_ctx.cpp:1507) [11788][][T0][Y0-0000000000000000-0-0] [lt=5] [STAT] slow fetch progress idx=0/1 tls_id={tenant_id:1012, ls_id:{id:1}} mod=IDLE_POOL discarded=0 delay=[20:04:01.221402] tps=0.00 progress={next_lsn:{lsn:18446744073709551615}, log_progress:"[2025-08-27 20:21:56.397904]", log_touch_tstamp:"[2025-08-28 15:59:11.526024]"}
395 [2025-08-28 16:25:57.619331] INFO  [TLOG.FETCHER] print_k_slowest_ls (ob_log_ls_fetch_mgr.cpp:436) [11788][][T0][Y0-0000000000000000-0-0] [lt=9] [STAT] slow fetch progress end
396 [2025-08-28 16:25:57.619336] INFO  [TLOG.FETCHER] print_k_slowest_ls (ob_log_ls_fetch_mgr.cpp:441) [11788][][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] slow dispatch progress start. ls_count=1/1
397 [2025-08-28 16:25:57.619344] INFO  [TLOG.FETCHER] print_dispatch_progress (ob_log_ls_fetch_ctx.cpp:1529) [11788][][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] slow dispatch progress idx=0/1 tls_id={tenant_id:1012, ls_id:{id:1}} delay=[20:04:01.221403] pending_task(queue/total)=0/0 dispatch_progress=[2025-08-27 20:21:56.397903] last_dispatch_log_lsn=18446744073709551615 next_task=INVALID next_trans(log_lsn=18446744073709551615,committed=0,ready_to_commit=0,global_version=[1970-01-01 08:00:00.000000]) checkpoint=[1970-01-01 08:00:00.000000]
398 [2025-08-28 16:25:57.619351] INFO  [TLOG.FETCHER] print_k_slowest_ls (ob_log_ls_fetch_mgr.cpp:448) [11788][][T0][Y0-0000000000000000-0-0] [lt=6] [STAT] slow dispatch progress end
399 [2025-08-28 16:25:57.619357] INFO  [TLOG.PARSER] stat_ignored_tps (ob_log_cluster_id_filter.cpp:130) [11788][][T0][Y0-0000000000000000-0-0] [lt=3] [TPS_STAT] CLUSTER_ID_IGNORED_PART_TPS=0.000
400 [2025-08-28 16:25:57.650057] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
401 [2025-08-28 16:25:57.650086] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F50-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F50-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F50-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
402 [2025-08-28 16:25:57.650123] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
403 [2025-08-28 16:25:57.650138] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=14][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F50-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F50-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F50-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
404 [2025-08-28 16:25:57.750243] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
405 [2025-08-28 16:25:57.750298] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
406 [2025-08-28 16:25:57.752541] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F51-0-0] [lt=24] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
407 [2025-08-28 16:25:57.752580] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F51-0-0] [lt=35] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
408 [2025-08-28 16:25:57.754662] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F51-0-0] [lt=24] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
409 [2025-08-28 16:25:57.754707] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F51-0-0] [lt=43] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
410 [2025-08-28 16:25:57.759065] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F51-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
411 [2025-08-28 16:25:57.759088] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F51-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
412 [2025-08-28 16:25:57.850399] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
413 [2025-08-28 16:25:57.850431] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F51-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F51-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F51-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
414 [2025-08-28 16:25:57.850482] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=50][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
415 [2025-08-28 16:25:57.850488] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F51-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F51-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F51-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
416 [2025-08-28 16:25:57.950611] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=40] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
417 [2025-08-28 16:25:57.950680] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
418 [2025-08-28 16:25:57.955225] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F52-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
419 [2025-08-28 16:25:57.955263] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F52-0-0] [lt=34] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
420 [2025-08-28 16:25:57.960467] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F52-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
421 [2025-08-28 16:25:57.960502] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F52-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
422 [2025-08-28 16:25:57.963141] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F52-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
423 [2025-08-28 16:25:57.963169] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F52-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
424 [2025-08-28 16:25:58.050780] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
425 [2025-08-28 16:25:58.050827] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=46][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F52-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F52-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F52-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
426 [2025-08-28 16:25:58.050864] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
427 [2025-08-28 16:25:58.050871] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=6][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F52-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F52-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F52-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
428 [2025-08-28 16:25:58.150984] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=38] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
429 [2025-08-28 16:25:58.151051] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
430 [2025-08-28 16:25:58.153690] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F53-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
431 [2025-08-28 16:25:58.153727] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F53-0-0] [lt=34] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
432 [2025-08-28 16:25:58.158711] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F53-0-0] [lt=25] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
433 [2025-08-28 16:25:58.158743] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F53-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
434 [2025-08-28 16:25:58.162297] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F53-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
435 [2025-08-28 16:25:58.162327] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F53-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
436 [2025-08-28 16:25:58.217479] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
437 [2025-08-28 16:25:58.218321] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
438 [2025-08-28 16:25:58.218462] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
439 [2025-08-28 16:25:58.218811] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
440 [2025-08-28 16:25:58.251144] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
441 [2025-08-28 16:25:58.251169] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F53-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F53-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F53-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
442 [2025-08-28 16:25:58.251203] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=33][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
443 [2025-08-28 16:25:58.251207] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F53-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F53-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F53-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
444 [2025-08-28 16:25:58.281331] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1756369558281327, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0
445 [2025-08-28 16:25:58.327381] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1756369558327378, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007551, add_bytes: 0
446 [2025-08-28 16:25:58.351298] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
447 [2025-08-28 16:25:58.351344] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=27] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
448 [2025-08-28 16:25:58.354073] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F54-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
449 [2025-08-28 16:25:58.354102] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F54-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
450 [2025-08-28 16:25:58.356247] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F54-0-0] [lt=22] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
451 [2025-08-28 16:25:58.356277] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F54-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
452 [2025-08-28 16:25:58.358691] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F54-0-0] [lt=24] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
453 [2025-08-28 16:25:58.358710] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F54-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
454 [2025-08-28 16:25:58.437856] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
455 [2025-08-28 16:25:58.438463] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
456 [2025-08-28 16:25:58.438463] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
457 [2025-08-28 16:25:58.438481] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
458 [2025-08-28 16:25:58.451423] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
459 [2025-08-28 16:25:58.451456] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F54-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F54-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F54-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
460 [2025-08-28 16:25:58.451498] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=40][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
461 [2025-08-28 16:25:58.451503] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F54-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F54-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F54-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
462 [2025-08-28 16:25:58.466750] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=3/0, request done=6015/0, request doing=0/0)
463 [2025-08-28 16:25:58.467185] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=6015/0, request doing=0/0)
464 [2025-08-28 16:25:58.467377] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=3/0, request done=6015/0, request doing=0/0)
465 [2025-08-28 16:25:58.467464] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=3/0, request done=6015/0, request doing=0/0)
466 [2025-08-28 16:25:58.468457] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
467 [2025-08-28 16:25:58.468458] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
468 [2025-08-28 16:25:58.468477] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
469 [2025-08-28 16:25:58.468490] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
470 [2025-08-28 16:25:58.551593] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=20] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
471 [2025-08-28 16:25:58.551664] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=49] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
472 [2025-08-28 16:25:58.555259] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F55-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
473 [2025-08-28 16:25:58.555299] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F55-0-0] [lt=37] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
474 [2025-08-28 16:25:58.557460] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F55-0-0] [lt=20] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
475 [2025-08-28 16:25:58.557485] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F55-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
476 [2025-08-28 16:25:58.560481] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F55-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
477 [2025-08-28 16:25:58.560510] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F55-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
478 [2025-08-28 16:25:58.651764] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
479 [2025-08-28 16:25:58.651811] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=46][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F55-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F55-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F55-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
480 [2025-08-28 16:25:58.651846] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
481 [2025-08-28 16:25:58.651850] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F55-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F55-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F55-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
482 [2025-08-28 16:25:58.751928] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
483 [2025-08-28 16:25:58.751988] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=40] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
484 [2025-08-28 16:25:58.754051] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F56-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
485 [2025-08-28 16:25:58.754077] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F56-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
486 [2025-08-28 16:25:58.756303] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F56-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
487 [2025-08-28 16:25:58.756325] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F56-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
488 [2025-08-28 16:25:58.759548] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F56-0-0] [lt=20] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
489 [2025-08-28 16:25:58.759576] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F56-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
490 [2025-08-28 16:25:58.852093] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
491 [2025-08-28 16:25:58.852130] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F56-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F56-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F56-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
492 [2025-08-28 16:25:58.852160] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
493 [2025-08-28 16:25:58.852165] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F56-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F56-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F56-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
494 [2025-08-28 16:25:58.952250] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
495 [2025-08-28 16:25:58.952307] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
496 [2025-08-28 16:25:58.954489] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F57-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
497 [2025-08-28 16:25:58.954516] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F57-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
498 [2025-08-28 16:25:58.956528] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F57-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
499 [2025-08-28 16:25:58.956545] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F57-0-0] [lt=15] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
500 [2025-08-28 16:25:58.963440] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F57-0-0] [lt=28] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
501 [2025-08-28 16:25:58.963481] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F57-0-0] [lt=39] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
502 [2025-08-28 16:25:59.052417] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
503 [2025-08-28 16:25:59.052460] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F57-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F57-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F57-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
504 [2025-08-28 16:25:59.052491] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
505 [2025-08-28 16:25:59.052496] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F57-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F57-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F57-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
506 [2025-08-28 16:25:59.152586] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
507 [2025-08-28 16:25:59.152642] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
508 [2025-08-28 16:25:59.155546] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F58-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
509 [2025-08-28 16:25:59.155580] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F58-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
510 [2025-08-28 16:25:59.157536] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F58-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
511 [2025-08-28 16:25:59.157563] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F58-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
512 [2025-08-28 16:25:59.160027] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F58-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
513 [2025-08-28 16:25:59.160044] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F58-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
514 [2025-08-28 16:25:59.218030] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
515 [2025-08-28 16:25:59.218463] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
516 [2025-08-28 16:25:59.218463] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
517 [2025-08-28 16:25:59.218740] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
518 [2025-08-28 16:25:59.252733] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
519 [2025-08-28 16:25:59.252758] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F58-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F58-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F58-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
520 [2025-08-28 16:25:59.252800] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
521 [2025-08-28 16:25:59.252805] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F58-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F58-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F58-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
522 [2025-08-28 16:25:59.288947] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1756369559288943, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0
523 [2025-08-28 16:25:59.334987] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1756369559334984, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007606, add_bytes: 0
524 [2025-08-28 16:25:59.352870] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
525 [2025-08-28 16:25:59.352925] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
526 [2025-08-28 16:25:59.354889] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F59-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
527 [2025-08-28 16:25:59.354917] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F59-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
528 [2025-08-28 16:25:59.357166] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F59-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
529 [2025-08-28 16:25:59.357186] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F59-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
530 [2025-08-28 16:25:59.359660] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F59-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
531 [2025-08-28 16:25:59.359677] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F59-0-0] [lt=15] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
532 [2025-08-28 16:25:59.437467] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
533 [2025-08-28 16:25:59.438460] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
534 [2025-08-28 16:25:59.438468] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
535 [2025-08-28 16:25:59.438929] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
536 [2025-08-28 16:25:59.453015] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
537 [2025-08-28 16:25:59.453036] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=20][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F59-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F59-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F59-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
538 [2025-08-28 16:25:59.453065] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
539 [2025-08-28 16:25:59.453069] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F59-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F59-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F59-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
540 [2025-08-28 16:25:59.466777] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=3/0, request done=6019/0, request doing=0/0)
541 [2025-08-28 16:25:59.466911] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6019/0, request doing=0/0)
542 [2025-08-28 16:25:59.467180] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=3/0, request done=6018/0, request doing=0/0)
543 [2025-08-28 16:25:59.467188] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6019/0, request doing=0/0)
544 [2025-08-28 16:25:59.468457] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
545 [2025-08-28 16:25:59.468457] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
546 [2025-08-28 16:25:59.468478] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
547 [2025-08-28 16:25:59.468491] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
548 [2025-08-28 16:25:59.553169] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
549 [2025-08-28 16:25:59.553236] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=46] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
550 [2025-08-28 16:25:59.555337] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F5A-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
551 [2025-08-28 16:25:59.555381] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F5A-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
552 [2025-08-28 16:25:59.557421] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F5A-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
553 [2025-08-28 16:25:59.557453] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F5A-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
554 [2025-08-28 16:25:59.560168] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F5A-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
555 [2025-08-28 16:25:59.560196] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F5A-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
556 [2025-08-28 16:25:59.653335] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
557 [2025-08-28 16:25:59.653383] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=47][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5A-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5A-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5A-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
558 [2025-08-28 16:25:59.653426] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
559 [2025-08-28 16:25:59.653430] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5A-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5A-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
560 [2025-08-28 16:25:59.753557] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
561 [2025-08-28 16:25:59.753610] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
562 [2025-08-28 16:25:59.756729] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F5B-0-0] [lt=33] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
563 [2025-08-28 16:25:59.756764] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F5B-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
564 [2025-08-28 16:25:59.760498] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F5B-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
565 [2025-08-28 16:25:59.760528] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F5B-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
566 [2025-08-28 16:25:59.763045] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F5B-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
567 [2025-08-28 16:25:59.763076] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F5B-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
568 [2025-08-28 16:25:59.853709] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
569 [2025-08-28 16:25:59.853747] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5B-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5B-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5B-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
570 [2025-08-28 16:25:59.853783] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
571 [2025-08-28 16:25:59.853788] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5B-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5B-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5B-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
572 [2025-08-28 16:25:59.953861] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
573 [2025-08-28 16:25:59.953937] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=52] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
574 [2025-08-28 16:25:59.957302] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F5C-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
575 [2025-08-28 16:25:59.957332] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F5C-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
576 [2025-08-28 16:25:59.959503] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F5C-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
577 [2025-08-28 16:25:59.959535] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F5C-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
578 [2025-08-28 16:25:59.962049] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F5C-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
579 [2025-08-28 16:25:59.962080] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F5C-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
580 [2025-08-28 16:26:00.054039] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
581 [2025-08-28 16:26:00.054070] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5C-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5C-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5C-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
582 [2025-08-28 16:26:00.054102] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
583 [2025-08-28 16:26:00.054107] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5C-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5C-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
584 [2025-08-28 16:26:00.154196] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
585 [2025-08-28 16:26:00.154249] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
586 [2025-08-28 16:26:00.156312] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F5D-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
587 [2025-08-28 16:26:00.156339] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F5D-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
588 [2025-08-28 16:26:00.158238] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F5D-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
589 [2025-08-28 16:26:00.158262] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F5D-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
590 [2025-08-28 16:26:00.160804] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F5D-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
591 [2025-08-28 16:26:00.160821] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F5D-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
592 [2025-08-28 16:26:00.218472] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
593 [2025-08-28 16:26:00.218472] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
594 [2025-08-28 16:26:00.218510] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
595 [2025-08-28 16:26:00.218994] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
596 [2025-08-28 16:26:00.254342] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
597 [2025-08-28 16:26:00.254379] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5D-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5D-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5D-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
598 [2025-08-28 16:26:00.254411] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
599 [2025-08-28 16:26:00.254427] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5D-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5D-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5D-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
600 [2025-08-28 16:26:00.296558] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1756369560296555, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0
601 [2025-08-28 16:26:00.342599] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1756369560342596, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0
602 [2025-08-28 16:26:00.354522] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=26] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
603 [2025-08-28 16:26:00.354563] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
604 [2025-08-28 16:26:00.356588] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F5E-0-0] [lt=26] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
605 [2025-08-28 16:26:00.356612] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F5E-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
606 [2025-08-28 16:26:00.358546] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F5E-0-0] [lt=25] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
607 [2025-08-28 16:26:00.358565] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F5E-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
608 [2025-08-28 16:26:00.361149] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F5E-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
609 [2025-08-28 16:26:00.361166] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F5E-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
610 [2025-08-28 16:26:00.438467] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
611 [2025-08-28 16:26:00.438467] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
612 [2025-08-28 16:26:00.438514] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
613 [2025-08-28 16:26:00.438518] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
614 [2025-08-28 16:26:00.454664] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
615 [2025-08-28 16:26:00.454688] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5E-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5E-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5E-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
616 [2025-08-28 16:26:00.454722] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=33][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
617 [2025-08-28 16:26:00.454727] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5E-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5E-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
618 [2025-08-28 16:26:00.466659] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=6023/0, request doing=0/0)
619 [2025-08-28 16:26:00.466868] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=3/0, request done=6022/0, request doing=0/0)
620 [2025-08-28 16:26:00.467221] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6023/0, request doing=0/0)
621 [2025-08-28 16:26:00.467705] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=3/0, request done=6022/0, request doing=0/0)
622 [2025-08-28 16:26:00.468456] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
623 [2025-08-28 16:26:00.468463] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
624 [2025-08-28 16:26:00.468470] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
625 [2025-08-28 16:26:00.468481] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
626 [2025-08-28 16:26:00.554812] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
627 [2025-08-28 16:26:00.554899] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=48] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
628 [2025-08-28 16:26:00.557015] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F5F-0-0] [lt=34] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
629 [2025-08-28 16:26:00.557041] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F5F-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
630 [2025-08-28 16:26:00.559072] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F5F-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
631 [2025-08-28 16:26:00.559092] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F5F-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
632 [2025-08-28 16:26:00.561849] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F5F-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
633 [2025-08-28 16:26:00.561881] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F5F-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
634 [2025-08-28 16:26:00.654991] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
635 [2025-08-28 16:26:00.655030] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=38][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5F-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5F-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5F-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
636 [2025-08-28 16:26:00.655062] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
637 [2025-08-28 16:26:00.655066] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F5F-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F5F-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F5F-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
638 [2025-08-28 16:26:00.755155] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
639 [2025-08-28 16:26:00.755212] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=38] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
640 [2025-08-28 16:26:00.757287] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F60-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
641 [2025-08-28 16:26:00.757336] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F60-0-0] [lt=46] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
642 [2025-08-28 16:26:00.759308] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F60-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
643 [2025-08-28 16:26:00.759340] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F60-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
644 [2025-08-28 16:26:00.761761] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F60-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
645 [2025-08-28 16:26:00.761782] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F60-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
646 [2025-08-28 16:26:00.855311] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
647 [2025-08-28 16:26:00.855350] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=38][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F60-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F60-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F60-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
648 [2025-08-28 16:26:00.855397] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=38][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
649 [2025-08-28 16:26:00.855402] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F60-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F60-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F60-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
650 [2025-08-28 16:26:00.955464] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
651 [2025-08-28 16:26:00.955516] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
652 [2025-08-28 16:26:00.957859] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F61-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
653 [2025-08-28 16:26:00.957885] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F61-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
654 [2025-08-28 16:26:00.959964] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F61-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
655 [2025-08-28 16:26:00.959982] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F61-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
656 [2025-08-28 16:26:00.962565] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F61-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
657 [2025-08-28 16:26:00.962582] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F61-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
658 [2025-08-28 16:26:01.055616] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
659 [2025-08-28 16:26:01.055650] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=33][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F61-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F61-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F61-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
660 [2025-08-28 16:26:01.055681] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
661 [2025-08-28 16:26:01.055685] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F61-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F61-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F61-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
662 [2025-08-28 16:26:01.155749] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
663 [2025-08-28 16:26:01.155809] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=39] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
664 [2025-08-28 16:26:01.157905] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F62-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
665 [2025-08-28 16:26:01.157931] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F62-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
666 [2025-08-28 16:26:01.160018] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F62-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
667 [2025-08-28 16:26:01.160037] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F62-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
668 [2025-08-28 16:26:01.162758] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F62-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
669 [2025-08-28 16:26:01.162780] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F62-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
670 [2025-08-28 16:26:01.218473] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
671 [2025-08-28 16:26:01.218473] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
672 [2025-08-28 16:26:01.218522] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
673 [2025-08-28 16:26:01.218540] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
674 [2025-08-28 16:26:01.255920] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=41][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
675 [2025-08-28 16:26:01.255952] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F62-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F62-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F62-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
676 [2025-08-28 16:26:01.255984] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
677 [2025-08-28 16:26:01.255988] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F62-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F62-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F62-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
678 [2025-08-28 16:26:01.304176] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1756369561304172, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0
679 [2025-08-28 16:26:01.350214] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1756369561350212, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0
680 [2025-08-28 16:26:01.356076] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
681 [2025-08-28 16:26:01.356117] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
682 [2025-08-28 16:26:01.358210] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F63-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
683 [2025-08-28 16:26:01.358234] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F63-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
684 [2025-08-28 16:26:01.360423] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F63-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
685 [2025-08-28 16:26:01.360453] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F63-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
686 [2025-08-28 16:26:01.362930] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F63-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
687 [2025-08-28 16:26:01.362953] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F63-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
688 [2025-08-28 16:26:01.438469] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
689 [2025-08-28 16:26:01.438469] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
690 [2025-08-28 16:26:01.438507] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
691 [2025-08-28 16:26:01.438533] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
692 [2025-08-28 16:26:01.456274] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
693 [2025-08-28 16:26:01.456300] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=25][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F63-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F63-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F63-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
694 [2025-08-28 16:26:01.456336] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
695 [2025-08-28 16:26:01.456342] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=6][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F63-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F63-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F63-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
696 [2025-08-28 16:26:01.467055] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=3/0, request done=6026/0, request doing=0/0)
697 [2025-08-28 16:26:01.467091] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=3/0, request done=6027/0, request doing=0/0)
698 [2025-08-28 16:26:01.467324] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=6026/0, request doing=0/0)
699 [2025-08-28 16:26:01.468468] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
700 [2025-08-28 16:26:01.468489] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
701 [2025-08-28 16:26:01.468506] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
702 [2025-08-28 16:26:01.469512] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
703 [2025-08-28 16:26:01.469540] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=6026/0, request doing=0/0)
704 [2025-08-28 16:26:01.556465] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
705 [2025-08-28 16:26:01.556516] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
706 [2025-08-28 16:26:01.558817] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F64-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
707 [2025-08-28 16:26:01.558847] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F64-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
708 [2025-08-28 16:26:01.560870] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F64-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
709 [2025-08-28 16:26:01.560895] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F64-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
710 [2025-08-28 16:26:01.563590] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F64-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
711 [2025-08-28 16:26:01.563613] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F64-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
712 [2025-08-28 16:26:01.619865] INFO  [TLOG] add_ls_and_fetch_until_the_progress_is_reached (ob_log_meta_data_fetcher.cpp:195) [11573][][T0][Y0-0000000000000000-0-0] [lt=5] ObLogFetcher check_progress is less than end_timestamp(tenant_id=1012, cur_progress=1756297316397903999, end_tstamp_ns=1756367342000000000)
713 [2025-08-28 16:26:01.619904] INFO  [TLOG] print_stat_info (ob_log_task_pool.h:266) [11573][][T0][Y0-0000000000000000-0-0] [lt=38] [STAT] [TRANS_TASK_POOL] OUT=0/1000 FIXED=0/1000 DYNAMIC=0 PAGE_POOL=1000/1000
714 [2025-08-28 16:26:01.656608] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
715 [2025-08-28 16:26:01.656637] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F64-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F64-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F64-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
716 [2025-08-28 16:26:01.656694] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=56][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
717 [2025-08-28 16:26:01.656701] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=6][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F64-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F64-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F64-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
718 [2025-08-28 16:26:01.756789] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=17] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
719 [2025-08-28 16:26:01.756844] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
720 [2025-08-28 16:26:01.758943] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F65-0-0] [lt=25] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
721 [2025-08-28 16:26:01.758984] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F65-0-0] [lt=38] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
722 [2025-08-28 16:26:01.760961] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F65-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
723 [2025-08-28 16:26:01.760991] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F65-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
724 [2025-08-28 16:26:01.763512] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F65-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
725 [2025-08-28 16:26:01.763536] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F65-0-0] [lt=23] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
726 [2025-08-28 16:26:01.856945] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=27][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
727 [2025-08-28 16:26:01.856976] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F65-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F65-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F65-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
728 [2025-08-28 16:26:01.857017] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=40][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
729 [2025-08-28 16:26:01.857022] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=5][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F65-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F65-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F65-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
730 [2025-08-28 16:26:01.957108] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
731 [2025-08-28 16:26:01.957159] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
732 [2025-08-28 16:26:01.959108] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F66-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
733 [2025-08-28 16:26:01.959138] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F66-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
734 [2025-08-28 16:26:01.961118] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F66-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
735 [2025-08-28 16:26:01.961144] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F66-0-0] [lt=24] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
736 [2025-08-28 16:26:01.963778] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F66-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
737 [2025-08-28 16:26:01.963800] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F66-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
738 [2025-08-28 16:26:02.036456] INFO  [COMMON] generate_mod_stat_task (memory_dump.cpp:218) [11579][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null})
739 [2025-08-28 16:26:02.036497] INFO  [COMMON] handle (memory_dump.cpp:498) [11579][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=40] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null})
740 [2025-08-28 16:26:02.036548] INFO  [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [11579][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] update_check_range(min_check_version=159, max_check_version=160, global_version=161)
741 [2025-08-28 16:26:02.037190] INFO  handle (memory_dump.cpp:626) [11579][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] statistics: 
742 tenant_cnt: 3, max_chunk_cnt: 524288
743 tenant_id      ctx_id         chunk_cnt      label_cnt      segv_cnt       
744 1              0              2              2              0              
745 500            0              40             87             0              
746 500            8              13             1              0              
747 500            17             9              1              0              
748 500            23             4              4              0              
749 cost_time: 642
750 [2025-08-28 16:26:02.057249] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
751 [2025-08-28 16:26:02.057272] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F66-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F66-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F66-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
752 [2025-08-28 16:26:02.057302] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
753 [2025-08-28 16:26:02.057306] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F66-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F66-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F66-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
754 [2025-08-28 16:26:02.157403] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=26] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
755 [2025-08-28 16:26:02.157477] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=52] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
756 [2025-08-28 16:26:02.159500] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F67-0-0] [lt=23] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
757 [2025-08-28 16:26:02.159524] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F67-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
758 [2025-08-28 16:26:02.161475] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F67-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
759 [2025-08-28 16:26:02.161493] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F67-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
760 [2025-08-28 16:26:02.163913] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F67-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
761 [2025-08-28 16:26:02.163931] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F67-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
762 [2025-08-28 16:26:02.218479] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
763 [2025-08-28 16:26:02.218482] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
764 [2025-08-28 16:26:02.218542] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
765 [2025-08-28 16:26:02.218582] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
766 [2025-08-28 16:26:02.257554] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=39][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
767 [2025-08-28 16:26:02.257587] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F67-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F67-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F67-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
768 [2025-08-28 16:26:02.257622] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=34][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
769 [2025-08-28 16:26:02.257627] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F67-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F67-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F67-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
770 [2025-08-28 16:26:02.311792] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1756369562311787, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0
771 [2025-08-28 16:26:02.357721] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
772 [2025-08-28 16:26:02.357793] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=50] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
773 [2025-08-28 16:26:02.357843] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1756369562357841, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007629, add_bytes: 0
774 [2025-08-28 16:26:02.359919] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F68-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
775 [2025-08-28 16:26:02.359948] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F68-0-0] [lt=27] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
776 [2025-08-28 16:26:02.362049] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F68-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
777 [2025-08-28 16:26:02.362071] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F68-0-0] [lt=20] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
778 [2025-08-28 16:26:02.364544] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F68-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
779 [2025-08-28 16:26:02.364572] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F68-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
780 [2025-08-28 16:26:02.438475] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
781 [2025-08-28 16:26:02.438475] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
782 [2025-08-28 16:26:02.438525] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
783 [2025-08-28 16:26:02.438542] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
784 [2025-08-28 16:26:02.457860] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
785 [2025-08-28 16:26:02.457885] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F68-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F68-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F68-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
786 [2025-08-28 16:26:02.457921] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
787 [2025-08-28 16:26:02.457927] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=6][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F68-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F68-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F68-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
788 [2025-08-28 16:26:02.466943] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=3/0, request done=6030/0, request doing=0/0)
789 [2025-08-28 16:26:02.467021] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=3/0, request done=6030/0, request doing=0/0)
790 [2025-08-28 16:26:02.467163] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=3/0, request done=6030/0, request doing=0/0)
791 [2025-08-28 16:26:02.467645] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=6030/0, request doing=0/0)
792 [2025-08-28 16:26:02.468461] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
793 [2025-08-28 16:26:02.468469] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
794 [2025-08-28 16:26:02.468485] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
795 [2025-08-28 16:26:02.468495] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
796 [2025-08-28 16:26:02.558029] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
797 [2025-08-28 16:26:02.558082] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=33] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
798 [2025-08-28 16:26:02.560190] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F69-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
799 [2025-08-28 16:26:02.560224] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F69-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
800 [2025-08-28 16:26:02.562661] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F69-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
801 [2025-08-28 16:26:02.562680] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F69-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
802 [2025-08-28 16:26:02.565276] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F69-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
803 [2025-08-28 16:26:02.565295] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F69-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
804 [2025-08-28 16:26:02.597984] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=14] [STAT] [ALL_ZONE] INDEX=0/3 ZONE=zone1 REGION=sys_region STORAGE_TYPE=LOCAL VERSION=322
805 [2025-08-28 16:26:02.598011] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=24] [STAT] [ALL_ZONE] INDEX=1/3 ZONE=zone2 REGION=sys_region STORAGE_TYPE=LOCAL VERSION=322
806 [2025-08-28 16:26:02.598018] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:425) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=6] [STAT] [ALL_ZONE] INDEX=2/3 ZONE=zone3 REGION=sys_region STORAGE_TYPE=LOCAL VERSION=322
807 [2025-08-28 16:26:02.598023] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_ZONE] INDEX=0/3 ZONE=zone1 ZONE_TYPE=ReadWrite VERSION=322
808 [2025-08-28 16:26:02.598028] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_ZONE] INDEX=1/3 ZONE=zone2 ZONE_TYPE=ReadWrite VERSION=322
809 [2025-08-28 16:26:02.598040] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:450) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=11] [STAT] [ALL_ZONE] INDEX=2/3 ZONE=zone3 ZONE_TYPE=ReadWrite VERSION=322
810 [2025-08-28 16:26:02.598044] INFO  [TLOG] update_zone_cache_ (ob_log_all_svr_cache.cpp:454) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] COUNT=3 VERSION=322
811 [2025-08-28 16:26:02.598053] INFO  [TLOG] purge_stale_zone_records_ (ob_log_all_svr_cache.cpp:660) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_ZONE] [PURGE] PURGE_COUNT=0 CUR_COUNT=3 VERSION=322
812 [2025-08-28 16:26:02.600909] INFO  [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=0/3 SERVER_ID=2 SERVER="172.17.0.11:2882" STATUS=1(ACTIVE) ZONE=zone2 REGION=sys_region(LOW) VERSION=322
813 [2025-08-28 16:26:02.600923] INFO  [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=12] [STAT] [ALL_SERVER_LIST] INDEX=1/3 SERVER_ID=1 SERVER="172.17.0.3:2882" STATUS=1(ACTIVE) ZONE=zone1 REGION=sys_region(LOW) VERSION=322
814 [2025-08-28 16:26:02.600928] INFO  [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:521) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=4] [STAT] [ALL_SERVER_LIST] INDEX=2/3 SERVER_ID=3 SERVER="172.17.16.5:2882" STATUS=1(ACTIVE) ZONE=zone3 REGION=sys_region(LOW) VERSION=322
815 [2025-08-28 16:26:02.600931] INFO  [TLOG] update_server_cache_ (ob_log_all_svr_cache.cpp:525) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] [STAT] [ALL_SERVER_LIST] COUNT=3 VERSION=322
816 [2025-08-28 16:26:02.600937] INFO  [TLOG] purge_stale_records_ (ob_log_all_svr_cache.cpp:646) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=2] [STAT] [ALL_SERVER_LIST] [PURGE] PURGE_COUNT=0 CUR_COUNT=3 VERSION=322
817 [2025-08-28 16:26:02.607567] INFO  [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=3] add_server_or_update(next_svr_index=0, svr_items=[{server:"172.17.0.11:2882", ranges:[LSN:{56636424192, 56718797122}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}])
818 [2025-08-28 16:26:02.607583] INFO  [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=15] add_server_or_update(next_svr_index=0, svr_items=[{server:"172.17.0.11:2882", ranges:[LSN:{56636424192, 56718797122}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"172.17.0.3:2882", ranges:[LSN:{56636424192, 56718797244}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}])
819 [2025-08-28 16:26:02.607589] INFO  [TLOG.FETCHER] add_server_or_update (ob_ls_server_list.cpp:87) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] add_server_or_update(next_svr_index=0, svr_items=[{server:"172.17.0.11:2882", ranges:[LSN:{56636424192, 56718797122}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"172.17.0.3:2882", ranges:[LSN:{56636424192, 56718797244}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}, {server:"172.17.16.5:2882", ranges:[LSN:{56636424192, 56718797122}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}])
820 [2025-08-28 16:26:02.607597] INFO  [TLOG] get_ls_svr_list_ (ob_log_route_service.cpp:817) [11755][LogRouter][T0][Y0-0000000000000000-0-0] [lt=5] get_ls_svr_list_ done(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}}, svr_list={next_svr_index:0, svr_num:3, svr_items:[{server:"172.17.0.11:2882", ranges:[LSN:{56636424192, 56718797122}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"172.17.16.5:2882", ranges:[LSN:{56636424192, 56718797122}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:0]}, {server:"172.17.0.3:2882", ranges:[LSN:{56636424192, 56718797244}], priority:[is_meta_table:1 region:LOW, replica:FULL, is_leader:1]}]}, tg_id_=259)
821 [2025-08-28 16:26:02.658161] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
822 [2025-08-28 16:26:02.658193] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F69-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F69-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F69-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
823 [2025-08-28 16:26:02.658223] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
824 [2025-08-28 16:26:02.658228] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F69-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F69-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F69-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
825 [2025-08-28 16:26:02.758716] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
826 [2025-08-28 16:26:02.758767] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
827 [2025-08-28 16:26:02.762275] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F6A-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
828 [2025-08-28 16:26:02.762310] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F6A-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
829 [2025-08-28 16:26:02.764331] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F6A-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
830 [2025-08-28 16:26:02.764359] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F6A-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
831 [2025-08-28 16:26:02.767127] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F6A-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
832 [2025-08-28 16:26:02.767150] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F6A-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
833 [2025-08-28 16:26:02.858861] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=42][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
834 [2025-08-28 16:26:02.858892] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6A-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6A-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6A-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
835 [2025-08-28 16:26:02.858923] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
836 [2025-08-28 16:26:02.858928] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6A-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6A-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6A-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
837 [2025-08-28 16:26:02.958973] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
838 [2025-08-28 16:26:02.959033] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=40] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
839 [2025-08-28 16:26:02.961129] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F6B-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
840 [2025-08-28 16:26:02.961162] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F6B-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
841 [2025-08-28 16:26:02.963245] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F6B-0-0] [lt=19] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
842 [2025-08-28 16:26:02.963264] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F6B-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
843 [2025-08-28 16:26:02.965703] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F6B-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
844 [2025-08-28 16:26:02.965722] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F6B-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
845 [2025-08-28 16:26:03.059088] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
846 [2025-08-28 16:26:03.059119] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6B-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6B-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6B-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
847 [2025-08-28 16:26:03.059159] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=39][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
848 [2025-08-28 16:26:03.059164] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6B-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6B-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6B-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
849 [2025-08-28 16:26:03.159222] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
850 [2025-08-28 16:26:03.159274] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
851 [2025-08-28 16:26:03.161275] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F6C-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
852 [2025-08-28 16:26:03.161309] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F6C-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
853 [2025-08-28 16:26:03.163305] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F6C-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
854 [2025-08-28 16:26:03.163326] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F6C-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
855 [2025-08-28 16:26:03.165741] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F6C-0-0] [lt=7] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
856 [2025-08-28 16:26:03.165758] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F6C-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
857 [2025-08-28 16:26:03.218470] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
858 [2025-08-28 16:26:03.218500] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
859 [2025-08-28 16:26:03.218522] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
860 [2025-08-28 16:26:03.218537] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
861 [2025-08-28 16:26:03.259375] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
862 [2025-08-28 16:26:03.259407] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6C-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6C-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6C-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
863 [2025-08-28 16:26:03.259459] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=51][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
864 [2025-08-28 16:26:03.259464] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6C-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6C-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6C-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
865 [2025-08-28 16:26:03.319408] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1756369563319404, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0
866 [2025-08-28 16:26:03.359509] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
867 [2025-08-28 16:26:03.359558] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
868 [2025-08-28 16:26:03.361580] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F6D-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
869 [2025-08-28 16:26:03.361604] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F6D-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
870 [2025-08-28 16:26:03.363652] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F6D-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
871 [2025-08-28 16:26:03.363671] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F6D-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
872 [2025-08-28 16:26:03.365455] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1756369563365453, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0
873 [2025-08-28 16:26:03.366188] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F6D-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
874 [2025-08-28 16:26:03.366205] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F6D-0-0] [lt=16] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
875 [2025-08-28 16:26:03.438476] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
876 [2025-08-28 16:26:03.438525] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=42] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
877 [2025-08-28 16:26:03.438538] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
878 [2025-08-28 16:26:03.438552] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
879 [2025-08-28 16:26:03.459661] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=33][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
880 [2025-08-28 16:26:03.459689] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=27][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6D-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6D-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6D-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
881 [2025-08-28 16:26:03.459720] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
882 [2025-08-28 16:26:03.459724] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6D-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6D-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6D-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
883 [2025-08-28 16:26:03.466595] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6034/0, request doing=0/0)
884 [2025-08-28 16:26:03.466595] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=3/0, request done=6034/0, request doing=0/0)
885 [2025-08-28 16:26:03.466852] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=3/0, request done=6033/0, request doing=0/0)
886 [2025-08-28 16:26:03.467285] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6034/0, request doing=0/0)
887 [2025-08-28 16:26:03.468457] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
888 [2025-08-28 16:26:03.468466] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
889 [2025-08-28 16:26:03.468480] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
890 [2025-08-28 16:26:03.468855] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
891 [2025-08-28 16:26:03.559809] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
892 [2025-08-28 16:26:03.559868] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=39] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
893 [2025-08-28 16:26:03.561992] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F6E-0-0] [lt=23] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
894 [2025-08-28 16:26:03.562027] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F6E-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
895 [2025-08-28 16:26:03.564038] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F6E-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
896 [2025-08-28 16:26:03.564068] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F6E-0-0] [lt=29] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
897 [2025-08-28 16:26:03.566821] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F6E-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
898 [2025-08-28 16:26:03.566840] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F6E-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
899 [2025-08-28 16:26:03.659930] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
900 [2025-08-28 16:26:03.659967] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6E-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6E-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6E-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
901 [2025-08-28 16:26:03.660000] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=32][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
902 [2025-08-28 16:26:03.660004] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6E-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6E-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6E-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
903 [2025-08-28 16:26:03.760108] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=39] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
904 [2025-08-28 16:26:03.760166] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
905 [2025-08-28 16:26:03.762235] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F6F-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
906 [2025-08-28 16:26:03.762271] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F6F-0-0] [lt=33] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
907 [2025-08-28 16:26:03.764318] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F6F-0-0] [lt=8] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
908 [2025-08-28 16:26:03.764360] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F6F-0-0] [lt=39] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
909 [2025-08-28 16:26:03.767494] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F6F-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
910 [2025-08-28 16:26:03.767526] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F6F-0-0] [lt=30] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
911 [2025-08-28 16:26:03.860259] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
912 [2025-08-28 16:26:03.860295] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6F-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6F-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6F-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
913 [2025-08-28 16:26:03.860327] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
914 [2025-08-28 16:26:03.860331] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F6F-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F6F-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F6F-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
915 [2025-08-28 16:26:03.960432] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=26] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
916 [2025-08-28 16:26:03.960503] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=51] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
917 [2025-08-28 16:26:03.962613] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F70-0-0] [lt=17] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
918 [2025-08-28 16:26:03.962637] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F70-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
919 [2025-08-28 16:26:03.964849] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F70-0-0] [lt=14] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
920 [2025-08-28 16:26:03.964869] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F70-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
921 [2025-08-28 16:26:03.967397] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F70-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
922 [2025-08-28 16:26:03.967416] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F70-0-0] [lt=17] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
923 [2025-08-28 16:26:04.060596] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
924 [2025-08-28 16:26:04.060626] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F70-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F70-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F70-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
925 [2025-08-28 16:26:04.060656] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=29][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
926 [2025-08-28 16:26:04.060670] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=13][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F70-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F70-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F70-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
927 [2025-08-28 16:26:04.160774] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
928 [2025-08-28 16:26:04.160838] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=45] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
929 [2025-08-28 16:26:04.162912] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F71-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
930 [2025-08-28 16:26:04.162937] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F71-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
931 [2025-08-28 16:26:04.164917] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F71-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
932 [2025-08-28 16:26:04.164955] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F71-0-0] [lt=37] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
933 [2025-08-28 16:26:04.167567] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F71-0-0] [lt=7] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
934 [2025-08-28 16:26:04.167601] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F71-0-0] [lt=32] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
935 [2025-08-28 16:26:04.218217] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11743][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
936 [2025-08-28 16:26:04.218223] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11744][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
937 [2025-08-28 16:26:04.218467] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11745][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
938 [2025-08-28 16:26:04.218488] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11742][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
939 [2025-08-28 16:26:04.260935] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
940 [2025-08-28 16:26:04.260964] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F71-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F71-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F71-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
941 [2025-08-28 16:26:04.261003] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=37][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
942 [2025-08-28 16:26:04.261012] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=8][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F71-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F71-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F71-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
943 [2025-08-28 16:26:04.327028] INFO  eloop_run (eloop.c:143) [11725][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1756369564327024, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0
944 [2025-08-28 16:26:04.361118] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
945 [2025-08-28 16:26:04.361171] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=35] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
946 [2025-08-28 16:26:04.363317] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F72-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
947 [2025-08-28 16:26:04.363352] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F72-0-0] [lt=31] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
948 [2025-08-28 16:26:04.365302] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F72-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
949 [2025-08-28 16:26:04.365332] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F72-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
950 [2025-08-28 16:26:04.368169] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F72-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
951 [2025-08-28 16:26:04.368193] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F72-0-0] [lt=22] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
952 [2025-08-28 16:26:04.373064] INFO  eloop_run (eloop.c:143) [11729][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1756369564373061, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007608, add_bytes: 0
953 [2025-08-28 16:26:04.437829] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11747][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
954 [2025-08-28 16:26:04.438462] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11751][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
955 [2025-08-28 16:26:04.438486] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11753][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
956 [2025-08-28 16:26:04.438501] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11749][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
957 [2025-08-28 16:26:04.461261] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=20][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
958 [2025-08-28 16:26:04.461283] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=21][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F72-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F72-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F72-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
959 [2025-08-28 16:26:04.461323] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=39][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
960 [2025-08-28 16:26:04.461328] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F72-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F72-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F72-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
961 [2025-08-28 16:26:04.467265] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11758][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=3/0, request done=6038/0, request doing=0/0)
962 [2025-08-28 16:26:04.467405] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11757][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=3/0, request done=6038/0, request doing=0/0)
963 [2025-08-28 16:26:04.467408] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11760][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6037/0, request doing=0/0)
964 [2025-08-28 16:26:04.467758] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11759][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=3/0, request done=6037/0, request doing=0/0)
965 [2025-08-28 16:26:04.468276] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11764][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
966 [2025-08-28 16:26:04.468282] INFO  [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [11762][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
967 [2025-08-28 16:26:04.468459] INFO  [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [11766][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
968 [2025-08-28 16:26:04.468468] INFO  [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [11768][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0)
969 [2025-08-28 16:26:04.561399] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
970 [2025-08-28 16:26:04.561458] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
971 [2025-08-28 16:26:04.563541] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F73-0-0] [lt=28] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
972 [2025-08-28 16:26:04.563571] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F73-0-0] [lt=28] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
973 [2025-08-28 16:26:04.565606] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F73-0-0] [lt=10] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
974 [2025-08-28 16:26:04.565626] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F73-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
975 [2025-08-28 16:26:04.568058] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F73-0-0] [lt=12] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
976 [2025-08-28 16:26:04.568078] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F73-0-0] [lt=19] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
977 [2025-08-28 16:26:04.661562] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
978 [2025-08-28 16:26:04.661609] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=46][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F73-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F73-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F73-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
979 [2025-08-28 16:26:04.661641] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
980 [2025-08-28 16:26:04.661645] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=3][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F73-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F73-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F73-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
981 [2025-08-28 16:26:04.761730] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=16] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
982 [2025-08-28 16:26:04.761779] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=28] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
983 [2025-08-28 16:26:04.763815] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F74-0-0] [lt=15] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
984 [2025-08-28 16:26:04.763860] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F74-0-0] [lt=42] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
985 [2025-08-28 16:26:04.766061] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F74-0-0] [lt=9] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
986 [2025-08-28 16:26:04.766101] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F74-0-0] [lt=39] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
987 [2025-08-28 16:26:04.768662] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F74-0-0] [lt=16] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
988 [2025-08-28 16:26:04.768689] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F74-0-0] [lt=25] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)
989 [2025-08-28 16:26:04.861872] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file="ob_log_start_lsn_locator.cpp", line_no=928, info="request start lsn from all server fail")
990 [2025-08-28 16:26:04.861918] EDIAG [TLOG.FETCHER] get_result (ob_log_start_lsn_locator.cpp:928) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=45][errcode=-4016] request start lsn from all server fail(tls_id={tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns=1756297316397904000, svr_cnt=3, svr_list_consumed=3, result_svr_list_idx=-1, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F74-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F74-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F74-0-0}]) BACKTRACE:0x1860ef10 0x9c403b5 0x9d5df6d 0x9d5d99f 0x9d5d8e8 0x9bbaae7 0xc3ce420 0xc1f73e6 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
991 [2025-08-28 16:26:04.861949] ERROR issue_dba_error (ob_log.cpp:1875) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=30][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_log_ls_fetch_ctx.cpp", line_no=933, info="start lsn locate fail")
992 [2025-08-28 16:26:04.861953] EDIAG [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:933) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=4][errcode=0] start lsn locate fail(start_lsn_locate_req={state:2, tls_id:{tenant_id:1012, ls_id:{id:1}}, start_tstamp_ns:1756297316397904000, svr_cnt:3, svr_list_consumed:3, result_svr_list_idx:-1, cur_max_start_lsn:{lsn:18446744073709551615}, cur_max_start_log_tstamp:-1, succ_locate_count:0, svr_list:[{svr:"172.17.0.3:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000501F74-0-0}, {svr:"172.17.0.11:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-4233, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000601F74-0-0}, {svr:"172.17.16.5:2882", rpc_executed:true, rpc_err:0, svr_err:0, ls_err:-9026, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y2D09AC11200D-0000000000701F74-0-0}], fetching_mode:"Integrated Fetching Mode", archive_locate_rs:{start_lsn:{lsn:18446744073709551615}, loc_err:0}}) BACKTRACE:0x1860ef10 0x9c403b5 0x9bf96cd 0x9bf910f 0x9bf904c 0x9bf8e7d 0xc1f8e9f 0xc1f73fb 0xc18233a 0xc181723 0xc180b78 0xc25e86f 0x7f99eda1bea5 0x7f99ed035b0d
993 [2025-08-28 16:26:04.962038] INFO  [TLOG] get_server_array_for_locate_start_lsn (ob_log_route_service.cpp:600) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=15] LSSvrList get_server_array_for_locate_start_lsn succ(ret=0, ret="OB_SUCCESS", router_key={cluster_id:3, tenant_id:1012, ls_id:{id:1}})
994 [2025-08-28 16:26:04.962094] INFO  [TLOG.FETCHER] locate_lsn_ (ob_log_ls_fetch_ctx.cpp:908) [11777][][T0][Y2D09AC11200D-0000000000300001-0-0] [lt=36] start lsn locate request launched(tls_id={tenant_id:1012, ls_id:{id:1}}, is_start_tstamp=true, start_tstamp="[2025-08-27 20:21:56.397904]", svr_cnt=3, svr_list=[{svr:"172.17.0.3:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.0.11:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}, {svr:"172.17.16.5:2882", rpc_executed:false, rpc_err:0, svr_err:0, ls_err:0, start_lsn:{lsn:18446744073709551615}, start_log_tstamp:18446744073709551615, trace_id:Y0-0000000000000000-0-0}])
995 [2025-08-28 16:26:04.964273] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11771][][T0][Y2D09AC11200D-0000000000501F75-0-0] [lt=18] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.3:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
996 [2025-08-28 16:26:04.964297] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11771][][T0][Y2D09AC11200D-0000000000501F75-0-0] [lt=21] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.3:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=1, svr_cnt=3)
997 [2025-08-28 16:26:04.966263] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11772][][T0][Y2D09AC11200D-0000000000601F75-0-0] [lt=11] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.0.11:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-4233, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
998 [2025-08-28 16:26:04.966283] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11772][][T0][Y2D09AC11200D-0000000000601F75-0-0] [lt=18] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.0.11:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_ERR_OUT_OF_LOWER_BOUND", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=2, svr_cnt=3)
999 [2025-08-28 16:26:04.968734] INFO  [TLOG] req_start_lsn_by_tstamp (ob_log_rpc.cpp:107) [11770][][T0][Y2D09AC11200D-0000000000701F75-0-0] [lt=13] rpc: request start LSN by tstamp(ret=0, ret="OB_SUCCESS", tenant_id=1012, svr="172.17.16.5:2882", timeout=60000000, req={rpc_ver:1, param_count:1, params:[{ls_id:{id:1}, start_ts_ns:1756297316397904000}], client_id:{client_addr:"172.17.32.13:11529", client_pid:11529}, flag:0}, resp={rpc_ver:1, err:0, result_count:1, result:[{err:-9026, start_lsn:{lsn:18446744073709551615}, start_ts_ns:-1}]})
1000 [2025-08-28 16:26:04.968761] INFO  [TLOG.FETCHER] set_result (ob_log_start_lsn_locator.cpp:894) [11770][][T0][Y2D09AC11200D-0000000000701F75-0-0] [lt=26] start lsn locate request of one server is done(ret=0, ret="OB_SUCCESS", tls_id={tenant_id:1012, ls_id:{id:1}}, svr_identity="172.17.16.5:2882", start_lsn={lsn:18446744073709551615}, start_log_tstamp=-1, start_tstamp_ns=1756297316397904000, delta=-1756297316397904001, rpc_err="OB_SUCCESS", svr_err="OB_SUCCESS", ls_err="OB_INVALID_BACKUP_DEST", cur_max_start_lsn_={lsn:18446744073709551615}, cur_max_start_log_tstamp_=-1, result_svr_list_idx_=-1, succ_locate_count_=0, svr_list_consumed=3, svr_cnt=3)