日志归档一直处于BEGINNING状态,持续一个半多小时了

【 使用环境 】测试环境
【 OB or 其他组件 】OB
【 使用版本 】v4.2.1.1
【问题描述】配置好备份路径后,启动租户的日志归档,租户日志归档状态一直处于BEGINNING状态,如下:

MySQL [(none)]> SELECT * FROM oceanbase.CDB_OB_ARCHIVELOG;
+-----------+---------+----------+-------------+---------+-----------+---------------------+----------------------------+---------------------+----------------------------+------------+---------------+---------------+-----------------------+-----------+-------------+-------------+---------------------+--------------+----------------------+-------------------+---------------------+-----------------------------+----------------------+------------------------------+---------+----------------------------------------------------------------------+
| TENANT_ID | DEST_ID | ROUND_ID | INCARNATION | DEST_NO | STATUS    | START_SCN           | START_SCN_DISPLAY          | CHECKPOINT_SCN      | CHECKPOINT_SCN_DISPLAY     | COMPATIBLE | BASE_PIECE_ID | USED_PIECE_ID | PIECE_SWITCH_INTERVAL | UNIT_SIZE | COMPRESSION | INPUT_BYTES | INPUT_BYTES_DISPLAY | OUTPUT_BYTES | OUTPUT_BYTES_DISPLAY | COMPRESSION_RATIO | DELETED_INPUT_BYTES | DELETED_INPUT_BYTES_DISPLAY | DELETED_OUTPUT_BYTES | DELETED_OUTPUT_BYTES_DISPLAY | COMMENT | PATH                                                                 |
+-----------+---------+----------+-------------+---------+-----------+---------------------+----------------------------+---------------------+----------------------------+------------+---------------+---------------+-----------------------+-----------+-------------+-------------+---------------------+--------------+----------------------+-------------------+---------------------+-----------------------------+----------------------+------------------------------+---------+----------------------------------------------------------------------+
|      1002 |    1001 |        1 |           1 |       0 | BEGINNING | 1734926415266312602 | 2024-12-23 12:00:15.266312 | 1734926415266312602 | 2024-12-23 12:00:15.266312 |          1 |             1 |             1 |           86400000000 |         1 | none        |           0 | 0.00MB              |            0 | 0.00MB               |              0.00 |                   0 | 0.00MB                      |                    0 | 0.00MB                       |         | file:///opt/obbackup/bc5eb5bd6abb4aadae1834dfaf309cff/mysqlt/archive |
+-----------+---------+----------+-------------+---------+-----------+---------------------+----------------------------+---------------------+----------------------------+------------+---------------+---------------+-----------------------+-----------+-------------+-------------+---------------------+--------------+----------------------+-------------------+---------------------+-----------------------------+----------------------+------------------------------+---------+----------------------------------------------------------------------+

观察rootservice.log,发现有一部分WDIAG级别的日志,怎么看哪些日志是归档状态处于BEGINNING状态的关键日志:

[2024-12-23 13:51:48.705047] WDIAG [RS] build_disaster_ls_info (ob_disaster_recovery_info.cpp:457) [10071][RootBalance][T0][YB420A750F71-000629E65B819ABB-0-0] [lt=42][errcode=-4018] fail to filter not in member/learner list replicas and learner_with_flag replicas(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1004, ls_id:{id:1002}, replicas:[{modify_time_us:1734919037559345, create_time_us:1734517492366865, tenant_id:1004, ls_id:{id:1002}, server:"10.117.15.113:2882", sql_port:2881, role:2, member_list:[{server:"10.117.15.113:2882", timestamp:1}, {server:"10.117.15.114:2882", timestamp:1}, {server:"10.117.15.115:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1004, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1734919033865752, create_time_us:1734517492454571, tenant_id:1004, ls_id:{id:1002}, server:"10.117.15.114:2882", sql_port:2881, role:2, member_list:[{server:"10.117.15.113:2882", timestamp:1}, {server:"10.117.15.114:2882", timestamp:1}, {server:"10.117.15.115:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1006, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1734517493451593, create_time_us:1734517492451651, tenant_id:1004, ls_id:{id:1002}, server:"10.117.15.115:2882", sql_port:2881, role:2, member_list:[{server:"10.117.15.113:2882", timestamp:1}, {server:"10.117.15.114:2882", timestamp:1}, {server:"10.117.15.115:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1005, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, filter_readonly_replicas_with_flag=true)
...
[2024-12-23 13:48:10.235158] WDIAG [RS] register_job (ob_dbms_sched_job_master.cpp:644) [10077][DBMS_SCHEDULER][T0][YB420A750F71-000629E655F15987-0-0] [lt=11][errcode=-4016] fail to exec table_operator_.calc_execute_at(job_info, execute_at, delay, ignore_nextdate)(ret=-4016)
...
[2024-12-23 13:48:14.686688] WDIAG [SQL] do_parse_ls_archive_piece_summary_result_ (ob_archive_persist_helper.cpp:1242) [10556][T1001_ArchiveSv][T1001][YB420A750F71-000629E65B317497-0-0] [lt=33][errcode=-4152] fail to get column in row. (column_name="ls_id", ret=-4152)
...
[2024-12-23 13:48:14.987287] WDIAG [RS] do_ls_arbitration_service_task_ (ob_arbitration_service.cpp:730) [10674][T1001_ArbSer][T1001][YB420A750F71-000629E65AE17426-0-0] [lt=26][errcode=-4747] fail to get arbitration service info(ret=-4747, ret="OB_ARBITRATION_SERVICE_NOT_EXIST", arbitration_service_key=default)
...


先看看NFS,如果NFS没问题 可以用obdiag收集一下日志信息发出来看看
https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000000267130

可以参考楼上发的链接排查下,如果不能解决需要获取下trace日志

1.开启 Trace 功能
SET ob_enable_show_trace=ON;
2.执行SQL
3.获取SQL trace_id
SELECT last_trace_id() FROM DUAL;
4.登录对应 OBServer 节点,进入到日志文件所在目录
cd /home/admin/oceanbase/log
5.获取trace_id对应的日志
grep xxxxxxx observer.log --填写第3步获取的trace_id
grep xxxxxxx rootservice.log --填写第3步获取的trace_id

1 个赞

发现环境有两个可疑的点:

  1. 有一个OBServer挂掉了,把归档停掉,重启OBServer。
  2. 挂掉的OBServer 时钟比其他机器慢了150秒左右。

把上面两个问题解决后,重启集群后重新归档又好了。

要先配置好时钟同步,这个很重要。时钟时间 差异过大 会引发 出各种各种的问题