数据库日志归档正常但是全量备份报错

OceanBase 技术问题
#1

本地自建nfs路径，日志归档可以正常归档，但是全量备份使用该路径就会报错，删除路径重建也会，使用黑屏操作也是失败，使用ocp操作备份也不行，但是日志归档却是正常的。我该如何排查呢
(ROOTSERVICE)addr: 192.168.1.2:2882, result: -9120(cannot find object on object storage), trace_id: YB420A012EA1-0006300C870CB43A-0-0

log.txt (22.9 KB)

1 个赞
#3

麻烦问下是几个节点的集群？所有节点都挂载了NFS吗？
查下OB详细版本

show variables like '%version_comment%';

grep "YB420A012EA1-0006300C870CB43A-0-0" rootservice.log
grep "YB420A012EA1-0006300C870CB43A-0-0" observer.log

上述信息麻烦发下

另外建议使用obdiag巡检下集群

全量巡检 (最常用)

obdiag check run

https://www.oceanbase.com/docs/common-obdiag-cn-1000000002200479

1 个赞
#4

3节点

1 个赞
#5

image
image1037×29 2.12 KB

日志搜不到，下面是巡检结果，所有节点都挂在了nfs
a.txt (21.6 KB)

1 个赞
#6

巡检报告显示主要xfs need repair和数据和日志同盘，和备份失败没有直接关系，

1.麻烦查下

select * from cdb_ob_archivelog;
select svr_ip from dba_ob_servers where with_rootserver='yes';
show parameters like '%syslog_level%';

2.再次发起备份，将报错日志发下

3.将第1步查到的svr_ip服务器上的rootservice.log和observer.log打包发下

1 个赞
#7

log.zip (18.2 MB)
错误信息还是这个 但是这个ip咋和查的svr_ip不一样呢
(ROOTSERVICE)addr: 10.1.46.161:2882, result: -9120(cannot find object on object storage), trace_id: YB420A012EA1-0006300C870CBF7A-0-0

image
image1263×75 3.92 KB

image

image
image1241×106 5.89 KB

1 个赞
#8

ocp.log (16.4 KB)
这个是ocp日志

1 个赞
#9

10.1.46.161上的rootsevice.log和observer.log也麻烦发下（包含14:07的日志）

1 个赞
#10

161.zip (21.5 MB)

1 个赞
#11

看起来是文件丢失了，麻烦黑屏做下如下操作，看是否可以成功

1.Alter system noarchivelog;
2.等2分钟

3.alter system set log_archive_dest= ' ';
4.alter system set data_backup_dest= ' ';
5.alter system set log_archive_dest= 'LOCATION=file:///xxx';
6.alter system set data_backup_dest= 'file:///xxx';
7.Alter system archivelog;
8.alter system backup database;

[2025-03-12 14:07:06.215726] INFO  get_file_length (ob_storage.cpp:792) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=31] cannot get file length for not exist file(ret=-9120, uri=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/infos/meta_info/ls_attr_info.1.obbak)
[2025-03-12 14:07:06.215747] WDIAG inner_stat_ (ob_object_device.cpp:735) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=21][errcode=-9120] fail to get file length!(ret=-9120, uri=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/infos/meta_info/ls_attr_info.1.obbak, is_adaptive=false)
[2025-03-12 14:07:06.215758] WDIAG get_file_length (ob_backup_io_adapter.cpp:236) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=10][errcode=-9120] fail to get file length!(ret=-9120, ret="OB_OBJECT_NOT_EXIST", uri=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/infos/meta_info/ls_attr_info.1.obbak, storage_info={endpoint:"", access_id:"", extension:"", type:"FILE", checksum_type:0, max_iops:0, max_bandwidth:0, role_arn:0x7f550d9d0990, external_id:0x7f550d9d0a90}, device_guard={device_handle_:0x7f56d98601e0, uri_cstr_:f})
[2025-03-12 14:07:06.215781] INFO  [SHARE] read_single_file (ob_backup_store.cpp:362) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=20] file not exist.(ret=-9120, full_path=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/infos/meta_info/ls_attr_info.1.obbak)
[2025-03-12 14:07:06.215820] WDIAG [SHARE] read_ls_attr_info (ob_backup_data_store.cpp:556) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=39][errcode=-9120] failed to read single file(ret=-9120, full_path=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/infos/meta_info/ls_attr_info.1.obbak)
[2025-03-12 14:07:06.216993] INFO  [SHARE] get_backup_dest (ob_backup_connectivity.cpp:1276) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=7] success get backup dest(sql=SELECT authorization,extension FROM __all_backup_storage_info WHERE tenant_id = 1002 AND path = 'file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data' AND endpoint = '', tenant_id=1002, backup_dest={root_path:"file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data", storage_info:{endpoint:"", access_id:"", extension:"", type:"FILE", checksum_type:0, max_iops:0, max_bandwidth:0, role_arn:0x7f550d9d2990, external_id:0x7f550d9d2a90}})
[2025-03-12 14:07:06.218250] INFO  get_file_length (ob_storage.cpp:792) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=29] cannot get file length for not exist file(ret=-9120, uri=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/logstream_1/meta_info_turn_1_retry_0/ls_meta_info.obbak)
[2025-03-12 14:07:06.218274] WDIAG inner_stat_ (ob_object_device.cpp:735) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=24][errcode=-9120] fail to get file length!(ret=-9120, uri=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/logstream_1/meta_info_turn_1_retry_0/ls_meta_info.obbak, is_adaptive=false)
[2025-03-12 14:07:06.218284] WDIAG get_file_length (ob_backup_io_adapter.cpp:236) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=10][errcode=-9120] fail to get file length!(ret=-9120, ret="OB_OBJECT_NOT_EXIST", uri=file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/logstream_1/meta_info_turn_1_retry_0/ls_meta_info.obbak, storage_info={endpoint:"", access_id:"", extension:"", type:"FILE", checksum_type:0, max_iops:0, max_bandwidth:0, role_arn:0x7f550d9dc990, external_id:0x7f550d9dca90}, device_guard={device_handle_:0x7f56d98601e0, uri_cstr_:f})
[2025-03-12 14:07:06.218305] INFO  [STORAGE] read_ls_meta_info (ob_backup_extern_info_mgr.cpp:210) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=19] tablet to ls info file not exist.(ret=-9120, path={cur_pos:141, path:"file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data/backup_set_12_full/logstream_1/meta_info_turn_1_retry_0/ls_meta_info.obbak"})
[2025-03-12 14:07:06.218314] WDIAG [RS] sync_wait_backup_user_ls_scn_ (ob_backup_data_set_task_mgr.cpp:344) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=8][errcode=-9120] failed to read ls meta info(ret=-9120)
[2025-03-12 14:07:06.218324] WDIAG [RS] persist_ls_attr_info_ (ob_backup_data_set_task_mgr.cpp:285) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=7][errcode=-9120] failed to calc backup user ls scn(ret=-9120)
[2025-03-12 14:07:06.218334] WDIAG [RS] backup_sys_meta_ (ob_backup_data_set_task_mgr.cpp:488) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=9][errcode=-9120] fail to do persist ls task(ret=-9120)
[2025-03-12 14:07:06.218345] WDIAG [RS] process (ob_backup_data_set_task_mgr.cpp:153) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=9][errcode=-9120] fail to backup sys meta(ret=-9120, set_task_attr_={task_id:12, tenant_id:1002, incarnation_id:1, job_id:12, backup_set_id:12, start_ts:1741759598763213, end_ts:0, start_scn:{val:1741759611477171000, v:0}, end_scn:{val:0, v:0}, user_ls_start_scn:{val:0, v:0}, data_turn_id:0, meta_turn_id:1, minor_turn_id:1, major_turn_id:1, status:{status:6}, encryption_mode:0, passwd:"", stats:{input_bytes:4197668, output_bytes:3379715, tablet_count:713, finish_tablet_count:3, macro_block_count:898, finish_macro_block_count:2, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, backup_path:"file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data", retry_cnt:0, result:0, comment:""})
[2025-03-12 14:07:06.218368] WDIAG [RS] do_set_task_ (ob_backup_data_scheduler.cpp:1409) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=22][errcode=-9120] [DATA_BACKUP]failed to backup ls task(ret=-9120, set_task_mgr={meta_tenant_id:1001, set_task_attr:{task_id:12, tenant_id:1002, incarnation_id:1, job_id:12, backup_set_id:12, start_ts:1741759598763213, end_ts:0, start_scn:{val:1741759611477171000, v:0}, end_scn:{val:0, v:0}, user_ls_start_scn:{val:0, v:0}, data_turn_id:0, meta_turn_id:1, minor_turn_id:1, major_turn_id:1, status:{status:6}, encryption_mode:0, passwd:"", stats:{input_bytes:4197668, output_bytes:3379715, tablet_count:713, finish_tablet_count:3, macro_block_count:898, finish_macro_block_count:2, extra_bytes:0, finish_file_count:0, log_file_count:0, finish_log_file_count:0}, backup_path:"file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data", retry_cnt:0, result:0, comment:""}})
[2025-03-12 14:07:06.218398] WDIAG [RS] process (ob_backup_data_scheduler.cpp:1168) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=28][errcode=-9120] [DATA_BACKUP]failed to backup data(ret=-9120, job_attr_={job_id:12, tenant_id:1002, incarnation_id:1, backup_set_id:12, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:false, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data", description:"ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-03-12 14:06:38.0", start_ts:1741759598763213, end_ts:0, status:{status:1}, result:0, can_retry:true, retry_count:0, comment:""})
[2025-03-12 14:07:06.218421] WDIAG [RS] process (ob_backup_data_scheduler.cpp:993) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=22][errcode=0] [DATA_BACKUP]failed to schedule tenant backup job(tmp_ret=-9120, tenant_id=1001, job_attr={job_id:12, tenant_id:1002, incarnation_id:1, backup_set_id:12, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:false, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data", description:"ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-03-12 14:06:38.0", start_ts:1741759598763213, end_ts:0, status:{status:1}, result:0, can_retry:true, retry_count:0, comment:""})
[2025-03-12 14:07:06.218439] INFO  [RS] deal_non_reentrant_job (ob_backup_data_scheduler.cpp:1086) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=17] start to deal non reentrant job(job_attr_={job_id:12, tenant_id:1002, incarnation_id:1, backup_set_id:12, initiator_tenant_id:1, initiator_job_id:8, executor_tenant_id:[1002], plus_archivelog:false, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"file:///data/nfs/test/1741079861/tenant_incarnation_1/1002/data", description:"ALTER BACKUP CLUSTER FULL_BACKUP BY BACKUP_NOW AT 2025-03-12 14:06:38.0", start_ts:1741759598763213, end_ts:1741759626218438, status:{status:1}, result:-9120, can_retry:true, retry_count:0, comment:""}, err=0xffffffffffffdc60)
[2025-03-12 14:07:06.221691] INFO  [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [19493][T1001_BackupDat][T1001][YB420A012EA1-0006300C870CBF7A-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1741759626221239000, start_id=1741759626221239000, end_id=1741759626221239001)
1 个赞
#12

再试试吧 这套操作我有搞过，我是直接连主ob还是继续通过obproxy连接呢

1 个赞
#13

(ROOTSERVICE)addr: 10.1.46.161:2882, result: -9120(cannot find object on object storage), trace_id: YB420A012EA1-0006300C870CC13F-0-0
一样的还是报这个

1 个赞
#14

直连OB

1 个赞
#15

不行，换了s3存储可以了 :sweat_smile:

#16

建议检查下NFS是否标准部署

部署 NFS

https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000002012862

#17

顺便问你个问题，这个4.3.5开源版支持对归档日志和全量备份数据压缩吗，我看归档日志运行情况里有这个但是为空，官方文档没找到设置压缩

image
image762×93 3.34 KB

#18

@论坛小助手

#19

根据文档库中的信息，OceanBase 数据库确实支持对归档日志和全量备份数据进行压缩。具体来说：

  1. 日志传输压缩：

    • OceanBase 数据库提供日志传输压缩功能，通过设置租户级配置项 log_transport_compress_all 控制 Redo 日志在网络传输中是否进行压缩。开启日志传输压缩可以减小 Redo 日志同步过程中的网络带宽占用。
    • 参见文档标题: Redo 日志 → 日志的压缩。

  2. 日志存储压缩：

    • 从 V4.2.1 版本开始重新支持日志的存储压缩功能（不含 V4.3.0 版本）。可以使用租户级配置项 log_storage_compress_all 来控制 Redo 日志在存储前是否进行压缩，开启日志存储压缩可以降低日志盘 IO 带宽的使用及日志盘空间的占用。
    • 参见文档标题: enable_clog_persistence_compress 和 clog_persistence_compress_func。

  3. 全量备份数据压缩：

    • 对于全量备份数据的压缩，虽然没有直接提到特定的配置项，但根据您提到的情况以及官方文档未明确说明，推测可能是依赖于底层存储系统的特性或默认启用。对于 NFS 存储路径出现问题而 S3 存储正常的现象，可能是因为 NFS 配置或兼容性问题导致压缩功能未能正确应用。

针对您的具体情况，建议如下：

  • 检查 NFS 配置：确保 NFS 挂载选项和权限配置正确，并且与 OceanBase 的要求相匹配。可以参考 NFS 部署指南。

  • 验证压缩配置：确认是否已经设置了相关的压缩参数如 log_storage_compress_all 和 log_transport_compress_all，并且这些参数值为 True 或者选择了合适的压缩算法。

  • 测试其他存储类型：既然 S3 存储能够正常工作，您可以暂时继续使用 S3 进行备份操作，同时排查 NFS 环境的问题。

如果您需要进一步的帮助或者具体的配置指导，请提供更多关于当前环境配置的信息，例如 NFS 挂载命令、OceanBase 版本等详细内容。这样可以帮助更准确地定位问题并给出解决方案。

更多信息请查看:

  1. enable_clog_persistence_compress
  2. Redo 日志
  3. clog_persistence_compress_func
  4. clog_transport_compress_func
  5. 备份架构
  6. 使用数据编码和压缩
  7. default_compress_func
  8. 表压缩
  9. 自动清理过期备份
  10. 体验并行导入和数据压缩