[2025-11-04 15:41:12.898133] INFO [SHARE] get_jobs (ob_backup_data_table_operator.cpp:711) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=18740] [DATA_BACKUP]success get jobs(job_attrs=[{job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:0, status:{status:1}, result:0, can_retry:true, retry_count:0, comment:""}]) [2025-11-04 15:41:12.898736] INFO [RS] do_set_task_ (ob_backup_data_scheduler.cpp:1392) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=400] job is in doing backup set task(job_attr_={job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:0, status:{status:1}, result:0, can_retry:true, retry_count:0, comment:""}) [2025-11-04 15:41:13.005262] WDIAG [RS] check_can_backup_ (ob_backup_data_scheduler.cpp:1315) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=35][errcode=-9027] [DATA_BACKUP]not supported backup when log archive is not doing(ret=-9027, round_attr={key:{tenant_id:1002, dest_no:0}, incarnation:1, dest_id:1012, round_id:6, state:{status:"INTERRUPTED"}, start_scn:{val:1762241657518941000, v:0}, checkpoint_scn:{val:1762241900477664001, v:0}, max_scn:{val:1762241901490294003, v:0}, compatible:{version:1}, base_piece_id:6, used_piece_id:6, piece_switch_interval:86400000000, frozen_input_bytes:0, frozen_output_bytes:0, active_input_bytes:93776439, active_output_bytes:93776439, deleted_input_bytes:0, deleted_output_bytes:0, path:"file:///hybackup/archive", comment:"log stream 1 interrupted"}) [2025-11-04 15:41:13.005320] WDIAG [RS] do_set_task_ (ob_backup_data_scheduler.cpp:1394) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=56][errcode=-9027] [DATA_BACKUP]failed to check can backup(ret=-9027, job_attr_={job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:0, status:{status:1}, result:0, can_retry:true, retry_count:0, comment:""}) [2025-11-04 15:41:13.005425] WDIAG [RS] process (ob_backup_data_scheduler.cpp:1161) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=103][errcode=-9027] [DATA_BACKUP]failed to backup data(ret=-9027, job_attr_={job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:0, status:{status:1}, result:0, can_retry:true, retry_count:0, comment:""}) [2025-11-04 15:41:13.005494] WDIAG [RS] process (ob_backup_data_scheduler.cpp:986) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=66][errcode=0] [DATA_BACKUP]failed to schedule tenant backup job(tmp_ret=-9027, tenant_id=1001, job_attr={job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:0, status:{status:1}, result:0, can_retry:true, retry_count:0, comment:""}) [2025-11-04 15:41:13.005521] INFO [RS] deal_non_reentrant_job (ob_backup_data_scheduler.cpp:1079) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=23] start to deal non reentrant job(job_attr_={job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:1762242073005519, status:{status:1}, result:-9027, can_retry:true, retry_count:0, comment:""}, err=0xffffffffffffdcbd) [2025-11-04 15:41:13.005545] INFO [RS] clean_tenant_ref_ (ob_backup_task_scheduler.cpp:743) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=19] tenant task cnt decrease(tenant_id=1002) [2025-11-04 15:41:13.005557] INFO [RS] remove_task_ (ob_backup_task_scheduler.cpp:1069) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=10] succeed remove task from backup scheduler mgr(task={ObBackupScheduleTask:{task_key:{tenant_id:1002, job_id:10, task_id:8, ls_id:1001, type:0}, trace_id:Y0-0000000000000000-0-0, dst:"0.0.0.0:0", status:{status:1}, optional_servers:[{server:"127.0.0.1:2882", priority:1}], generate_time:1762242012496355, schedule_time:0, executor_time:0, last_check_alive_time:1762242012493359}, incarnation_id:1, backup_set_id:8, backup_type:{type:1}, backup_date:20251104, ls_id:{id:1001}, turn_id:1, retry_id:2, start_scn:{val:1762241744923829000, v:0}, backup_user_ls_scn:{val:1762241810340978000, v:0}, end_scn:{val:0, v:0}, backup_path:"file:///hybackup/data", backup_status:{status:10}, fuse_turn_id:0}) [2025-11-04 15:41:13.018977] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=35][errcode=-4018] execute rpc fail(addr="127.0.0.1:2882", pcode=1079, ret=-4018, timeout=9000000) [2025-11-04 15:41:13.019030] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=48][errcode=-4018] [2025-11-04 15:41:13.019054] INFO [RS] cancel (ob_backup_schedule_task.cpp:367) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=13] task may not excute on server(rpc_arg={task_id:YB427F000001-000642BF88EACC69-0-0}, dst="127.0.0.1:2882") [2025-11-04 15:41:13.019081] INFO [RS] clean_server_ref_ (ob_backup_task_scheduler.cpp:721) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=20] server task cnt decrease(key={type:0, addr:"127.0.0.1:2882", hash_value:18061519644617349199}) [2025-11-04 15:41:13.019102] INFO [RS] clean_tenant_ref_ (ob_backup_task_scheduler.cpp:743) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=15] tenant task cnt decrease(tenant_id=1002) [2025-11-04 15:41:13.019114] INFO [RS] remove_task_ (ob_backup_task_scheduler.cpp:1069) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=8] succeed remove task from backup scheduler mgr(task={ObBackupScheduleTask:{task_key:{tenant_id:1002, job_id:10, task_id:8, ls_id:1, type:0}, trace_id:YB427F000001-000642BF88EACC69-0-0, dst:"127.0.0.1:2882", status:{status:2}, optional_servers:[{server:"127.0.0.1:2882", priority:1}], generate_time:1762242012409643, schedule_time:1762242012510810, executor_time:1762242012521549, last_check_alive_time:1762242072977673}, incarnation_id:1, backup_set_id:8, backup_type:{type:1}, backup_date:20251104, ls_id:{id:1}, turn_id:1, retry_id:2, start_scn:{val:1762241744923829000, v:0}, backup_user_ls_scn:{val:1762241744923829000, v:0}, end_scn:{val:0, v:0}, backup_path:"file:///hybackup/data", backup_status:{status:10}, fuse_turn_id:0}) [2025-11-04 15:41:13.039137] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=126] get number(ret=0, service_type_=0, range=1, base_id=1762242073023251000, start_id=1762242073023251000, end_id=1762242073023251001) [2025-11-04 15:41:13.066519] INFO [SHARE] update_comment (ob_backup_data_table_operator.cpp:1079) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=128] [DATA_BACKUP]update comment(job_attr={job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:1762242073005519, status:{status:1}, result:-9027, can_retry:true, retry_count:0, comment:"(ROOTSERVICE)addr: 127.0.0.1:2882, result: -9027(ob log archive interrupted), trace_id: YB427F000001-000642BF890ACCB5-0-0"}) [2025-11-04 15:41:13.112884] INFO [SHARE] get_backup_task (ob_backup_data_table_operator.cpp:1396) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=58] [DATA_BACKUP]succeed to get set task(set_task_attr={task_id:8, tenant_id:1002, incarnation_id:1, job_id:10, backup_set_id:8, start_ts:1762241740659325, end_ts:0, start_scn:{val:1762241744923829000, v:0}, end_scn:{val:0, v:0}, user_ls_start_scn:{val:1762241810340978000, v:0}, data_turn_id:0, meta_turn_id:1, minor_turn_id:1, major_turn_id:1, status:{status:10}, encryption_mode:0, passwd:"", stats:{input_bytes:6297815, output_bytes:1954761, tablet_count:1209, finish_tablet_count:6, macro_block_count:1465, finish_macro_block_count:3, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, backup_path:"file:///hybackup/data", retry_cnt:0, result:0, comment:""}) [2025-11-04 15:41:13.118088] INFO [SHARE] get_backup_dest (ob_backup_connectivity.cpp:1268) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=142] success get backup dest(sql=SELECT authorization,extension FROM __all_backup_storage_info WHERE tenant_id = 1002 AND path = 'file:///hybackup/data' AND endpoint = '', tenant_id=1002, backup_dest={root_path:"file:///hybackup/data", storage_info:{endpoint:"", access_id:"", extension:"", type:"FILE", checksum_type:0, max_iops:0, max_bandwidth:0, role_arn:0x147def4d8f90, external_id:0x147def4d9090}}) [2025-11-04 15:41:13.139002] INFO [SHARE] do_parse_ls_result_ (ob_backup_data_table_operator.cpp:2011) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=52] [DATA_BACKUP]success to read ls attr(ls_attr={task_id:8, tenant_id:1002, ls_id:{id:1}, job_id:10, backup_set_id:8, backup_type:{type:1}, task_type:{type:2}, status:{status:2}, start_ts:1762241746755265, end_ts:1762241980964063, backup_date:20251104, black_servers:["127.0.0.1:2882"], dst:"127.0.0.1:2882", task_trace_id:YB427F000001-000642BF88EACC69-0-0, stats:{input_bytes:2099929, output_bytes:1655246, tablet_count:725, finish_tablet_count:3, macro_block_count:833, finish_macro_block_count:1, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:2, result:0, comment:"(SERVER)ls_id: 1, addr: 127.0.0.1:2882, module: BACKUP_DATA, result: -4392(disk is hung), trace_id: YB427F000001-000642BF88EACC67-0-0", max_tablet_checkpoint_scn:{val:1762241750182371000, v:0}}) [2025-11-04 15:41:13.139106] INFO [SHARE] do_parse_ls_result_ (ob_backup_data_table_operator.cpp:2011) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=84] [DATA_BACKUP]success to read ls attr(ls_attr={task_id:8, tenant_id:1002, ls_id:{id:1001}, job_id:10, backup_set_id:8, backup_type:{type:1}, task_type:{type:2}, status:{status:1}, start_ts:1762241810445090, end_ts:1762241982016402, backup_date:20251104, black_servers:["127.0.0.1:2882"], dst:"0.0.0.0:0", task_trace_id:Y0-0000000000000000-0-0, stats:{input_bytes:4197886, output_bytes:299515, tablet_count:484, finish_tablet_count:3, macro_block_count:632, finish_macro_block_count:2, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:2, result:0, comment:"(SERVER)ls_id: 1001, addr: 127.0.0.1:2882, module: BACKUP_DATA, result: -4392(disk is hung), trace_id: YB427F000001-000642BF88EACC68-0-0", max_tablet_checkpoint_scn:{val:1762241817937346001, v:0}}) [2025-11-04 15:41:13.139139] INFO [SHARE] get_ls_tasks (ob_backup_data_table_operator.cpp:1799) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=26] succ to get ls tasks(ls_attrs=[{task_id:8, tenant_id:1002, ls_id:{id:1}, job_id:10, backup_set_id:8, backup_type:{type:1}, task_type:{type:2}, status:{status:2}, start_ts:1762241746755265, end_ts:1762241980964063, backup_date:20251104, black_servers:["127.0.0.1:2882"], dst:"127.0.0.1:2882", task_trace_id:YB427F000001-000642BF88EACC69-0-0, stats:{input_bytes:2099929, output_bytes:1655246, tablet_count:725, finish_tablet_count:3, macro_block_count:833, finish_macro_block_count:1, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:2, result:0, comment:"(SERVER)ls_id: 1, addr: 127.0.0.1:2882, module: BACKUP_DATA, result: -4392(disk is hung), trace_id: YB427F000001-000642BF88EACC67-0-0", max_tablet_checkpoint_scn:{val:1762241750182371000, v:0}}, {task_id:8, tenant_id:1002, ls_id:{id:1001}, job_id:10, backup_set_id:8, backup_type:{type:1}, task_type:{type:2}, status:{status:1}, start_ts:1762241810445090, end_ts:1762241982016402, backup_date:20251104, black_servers:["127.0.0.1:2882"], dst:"0.0.0.0:0", task_trace_id:Y0-0000000000000000-0-0, stats:{input_bytes:4197886, output_bytes:299515, tablet_count:484, finish_tablet_count:3, macro_block_count:632, finish_macro_block_count:2, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:2, result:0, comment:"(SERVER)ls_id: 1001, addr: 127.0.0.1:2882, module: BACKUP_DATA, result: -4392(disk is hung), trace_id: YB427F000001-000642BF88EACC68-0-0", max_tablet_checkpoint_scn:{val:1762241817937346001, v:0}}]) [2025-11-04 15:41:13.160167] INFO [SHARE] report_ls_task (ob_backup_data_table_operator.cpp:1649) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=42] [DATA_BACKUP]update one ls task(ls_attr={task_id:8, tenant_id:1002, ls_id:{id:1}, job_id:10, backup_set_id:8, backup_type:{type:1}, task_type:{type:2}, status:{status:3}, start_ts:1762241746755265, end_ts:1762242073005519, backup_date:20251104, black_servers:["127.0.0.1:2882"], dst:"127.0.0.1:2882", task_trace_id:YB427F000001-000642BF88EACC69-0-0, stats:{input_bytes:2099929, output_bytes:1655246, tablet_count:725, finish_tablet_count:3, macro_block_count:833, finish_macro_block_count:1, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:2, result:0, comment:"(SERVER)ls_id: 1, addr: 127.0.0.1:2882, module: BACKUP_DATA, result: -4392(disk is hung), trace_id: YB427F000001-000642BF88EACC67-0-0", max_tablet_checkpoint_scn:{val:1762241750182371000, v:0}}, sql=UPDATE __all_backup_ls_task SET job_id = 10, backup_set_id = 8, backup_type = 'FULL', task_type = 'BACKUP_USER_DATA', status = 'FINISH', start_ts = 1762241746755265, end_ts = 1762242073005519, date = 20251104, black_list = '127.0.0.1:2882', svr_ip = '127.0.0.1', svr_port = 2882, task_trace_id = 'YB427F000001-000642BF88EACC69-0-0', output_bytes = 1655246, input_bytes = 2099929, tablet_count = 725, finish_tablet_count = 3, macro_block_count = 833, finish_macro_block_count = 1, extra_bytes = 0, file_count = 0, start_turn_id = 1, turn_id = 1, retry_id = 2, result = 0, comment = '(SERVER)ls_id: 1, addr: 127.0.0.1:2882, module: BACKUP_DATA, result: -4392(disk is hung), trace_id: YB427F000001-000642BF88EACC67-0-0', max_tablet_checkpoint_scn = 1762241750182371000, log_file_count = 0, finish_log_file_count = 0 WHERE task_id = 8 AND tenant_id = 1002 AND ls_id = 1, affected_rows=1) [2025-11-04 15:41:13.160765] INFO [SHARE] report_ls_task (ob_backup_data_table_operator.cpp:1649) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=110] [DATA_BACKUP]update one ls task(ls_attr={task_id:8, tenant_id:1002, ls_id:{id:1001}, job_id:10, backup_set_id:8, backup_type:{type:1}, task_type:{type:2}, status:{status:3}, start_ts:1762241810445090, end_ts:1762242073005519, backup_date:20251104, black_servers:["127.0.0.1:2882"], dst:"0.0.0.0:0", task_trace_id:Y0-0000000000000000-0-0, stats:{input_bytes:4197886, output_bytes:299515, tablet_count:484, finish_tablet_count:3, macro_block_count:632, finish_macro_block_count:2, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_turn_id:1, turn_id:1, retry_id:2, result:0, comment:"(SERVER)ls_id: 1001, addr: 127.0.0.1:2882, module: BACKUP_DATA, result: -4392(disk is hung), trace_id: YB427F000001-000642BF88EACC68-0-0", max_tablet_checkpoint_scn:{val:1762241817937346001, v:0}}, sql=UPDATE __all_backup_ls_task SET job_id = 10, backup_set_id = 8, backup_type = 'FULL', task_type = 'BACKUP_USER_DATA', status = 'FINISH', start_ts = 1762241810445090, end_ts = 1762242073005519, date = 20251104, black_list = '127.0.0.1:2882', svr_ip = '', svr_port = 0, task_trace_id = '', output_bytes = 299515, input_bytes = 4197886, tablet_count = 484, finish_tablet_count = 3, macro_block_count = 632, finish_macro_block_count = 2, extra_bytes = 0, file_count = 0, start_turn_id = 1, turn_id = 1, retry_id = 2, result = 0, comment = '(SERVER)ls_id: 1001, addr: 127.0.0.1:2882, module: BACKUP_DATA, result: -4392(disk is hung), trace_id: YB427F000001-000642BF88EACC68-0-0', max_tablet_checkpoint_scn = 1762241817937346001, log_file_count = 0, finish_log_file_count = 0 WHERE task_id = 8 AND tenant_id = 1002 AND ls_id = 1001, affected_rows=1) [2025-11-04 15:41:13.160953] INFO [SHARE] get_backup_dest (ob_backup_connectivity.cpp:1268) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=48] success get backup dest(sql=SELECT authorization,extension FROM __all_backup_storage_info WHERE tenant_id = 1002 AND path = 'file:///hybackup/data' AND endpoint = '', tenant_id=1002, backup_dest={root_path:"file:///hybackup/data", storage_info:{endpoint:"", access_id:"", extension:"", type:"FILE", checksum_type:0, max_iops:0, max_bandwidth:0, role_arn:0x147def4d8f90, external_id:0x147def4d9090}}) [2025-11-04 15:41:13.178637] INFO [SHARE] get_backup_set_file (ob_backup_data_table_operator.cpp:278) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=132] [DATA_BACKUP]get_backup_set_file(ret=0, sql=select * from __all_backup_set_files where backup_set_id=8 and incarnation=1 and tenant_id=1002 and dest_id=1013 for update, backup_set_desc={backup_set_id:8, incarnation:1, tenant_id:1002, dest_id:1013, backup_type:{type:1}, plus_archivelog:true, date:20251104, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, stats:{input_bytes:0, output_bytes:0, tablet_count:0, finish_tablet_count:0, macro_block_count:0, finish_macro_block_count:0, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_time:1762241740659325, end_time:0, status:0, result:0, encryption_mode:0, passwd:"", file_status:1, backup_path:"file:///hybackup/data", start_replay_scn:{val:0, v:0}, min_restore_scn:{val:0, v:0}, tenant_compatible:17180067073, backup_compatible:4, data_turn_id:0, meta_turn_id:0, cluster_version:17180067073, consistent_scn:{val:1762241847642304000, v:0}}) [2025-11-04 15:41:13.192585] INFO [SHARE] update_backup_set_file (ob_backup_data_table_operator.cpp:432) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=151] [DATA_BACKUP]update backup set(backup_set_desc={backup_set_id:8, incarnation:1, tenant_id:1002, dest_id:1013, backup_type:{type:1}, plus_archivelog:true, date:20251104, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, stats:{input_bytes:6297815, output_bytes:1954761, tablet_count:1209, finish_tablet_count:6, macro_block_count:1465, finish_macro_block_count:3, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, start_time:1762241740659325, end_time:1762242073005519, status:2, result:-9027, encryption_mode:0, passwd:"", file_status:2, backup_path:"file:///hybackup/data", start_replay_scn:{val:0, v:0}, min_restore_scn:{val:0, v:0}, tenant_compatible:17180067073, backup_compatible:4, data_turn_id:0, meta_turn_id:1, cluster_version:17180067073, consistent_scn:{val:1762241847642304000, v:0}}) [2025-11-04 15:41:13.199571] INFO [SHARE] advance_task_status (ob_backup_data_table_operator.cpp:1444) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=95] [DATA_BACKUP]advance status(set_task_attr={task_id:8, tenant_id:1002, incarnation_id:1, job_id:10, backup_set_id:8, start_ts:1762241740659325, end_ts:1762242073005519, start_scn:{val:1762241744923829000, v:0}, end_scn:{val:0, v:0}, user_ls_start_scn:{val:1762241810340978000, v:0}, data_turn_id:0, meta_turn_id:1, minor_turn_id:1, major_turn_id:1, status:{status:3}, encryption_mode:0, passwd:"", stats:{input_bytes:6297815, output_bytes:1954761, tablet_count:1209, finish_tablet_count:6, macro_block_count:1465, finish_macro_block_count:3, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, backup_path:"file:///hybackup/data", retry_cnt:0, result:0, comment:""}, next_status={status:3}, sql=UPDATE __all_backup_task SET status = 'FAILED', result = -9027, comment = 'ob log archive interrupted', end_ts = 1762242073005519, end_scn = 0 WHERE tenant_id = 1002 AND task_id = 8 AND job_id=10) [2025-11-04 15:41:13.219636] INFO [SHARE] add_event (ob_event_history_table_operator.h:266) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=110] event table add task(ret=0, event_table_name="__all_rootservice_event_history", sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1762242073199686), 'backup_data', 'advance_status', 'tenant_id', 1002, 'job_id', 10, 'backup_set_id', 8, 'curr_status', 'FAILED', 'next_status', 'FAILED', 'result', -9027, '127.0.0.1', 2882)) [2025-11-04 15:41:13.226825] INFO [SHARE] advance_job_status (ob_backup_data_table_operator.cpp:1002) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=253] [DATA_BACKUP]advance status(job_attr={job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:1762242073005519, status:{status:1}, result:-9027, can_retry:true, retry_count:0, comment:"(ROOTSERVICE)addr: 127.0.0.1:2882, result: -9027(ob log archive interrupted), trace_id: YB427F000001-000642BF890ACCB5-0-0"}, next_status={status:3}) [2025-11-04 15:41:13.256252] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=56] sql stat record not found(ret=-5138, key={sql_id:"1D0BA376E273B9D622641124D8C59264", plan_hash:4110590302725436204, source_addr:"0.0.0.0:0"}) [2025-11-04 15:41:13.256575] INFO [RS] deal_non_reentrant_job (ob_backup_data_scheduler.cpp:1130) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=29] succeed deal with failed job, advance job status to FAILED(job_attr_={job_id:10, tenant_id:1002, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:true, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///hybackup/data", description:"", start_ts:1762241740659325, end_ts:1762242073005519, status:{status:1}, result:-9027, can_retry:true, retry_count:0, comment:"(ROOTSERVICE)addr: 127.0.0.1:2882, result: -9027(ob log archive interrupted), trace_id: YB427F000001-000642BF890ACCB5-0-0"}) [2025-11-04 15:41:13.257003] INFO [SHARE] add_event (ob_event_history_table_operator.h:266) [512][T1001_BackupDat][T1001][YB427F000001-000642BF890ACCB5-0-0] [lt=371] event table add task(ret=0, event_table_name="__all_rootservice_event_history", sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1762242073256948), 'backup_data', 'deal with failed job', 'tenant_id', 1002, 'job_id', 10, 'result', -9027, 'trace_id', 'YB427F000001-000642BF890ACCB5-0-0', '', '', '127.0.0.1', 2882))