log_archive stoping状态卡住

OceanBase 技术问题
#1

生产环境 3.1.4-OceanBase CE ，5+5+1(log)架构
线上有个集群停止log archive后，长期处于stopping状态.
mysql> select gmt_create,gmt_modified,min_first_time,status from oceanbase.__all_backup_log_archive_status_v2;
±---------------------------±---------------------------±---------------------------±----------+
| gmt_create | gmt_modified | min_first_time | status |
±---------------------------±---------------------------±---------------------------±----------+
| 2022-08-29 11:26:56.187568 | 2023-06-09 21:57:47.256685 | 2023-05-26 05:18:15.128498 | STOPPING |
| 2023-05-26 05:18:15.673265 | 2023-05-26 05:18:15.673265 | 2022-08-29 14:22:55.891223 | BEGINNING |
| 2023-05-26 05:18:15.858877 | 2023-05-26 05:18:15.858877 | 2022-08-29 14:23:32.727618 | BEGINNING |
±---------------------------±---------------------------±---------------------------±----------+

#rs日志：
[2023-08-03 14:56:11.453512] ERROR [RS] check_mount_file_ (ob_log_archive_scheduler.cpp:1228) [131256][672][YB420A3AEEDF-0005F7EBC87B0215]
[lt=54] [dc=0] failed to check_mount_file(ret=-9051, sys_info={status:{tenant_id:1, copy_id:0, start_ts:1685049495128498, checkpoint_ts:0, status:4, incarnation:1, round:8, status_str:“STOPPING”, is_mark_deleted:false, is_mount_file_created:true, compatible:1, backup_piece_id:205, start_piece_id:205}, backup_dest:“file:///xxxx/commbackup/”})
BACKTRACE:0x9a98e9e 0x986d141 0x22e245f 0x22e20ab 0x22e1e72 0x38a8f2c 0x70d07cf 0x70ca92b 0x70c987b 0x6750a72 0x9a2aabd 0x9a2a4ee 0x340b9af 0x2cabf02 0x9820da5 0x981f792 0x981c24f

stoping状态已经有2个月了，
#1、尝试了 ALTER SYSTEM CANCEL ALL BACKUP FORCE; 无效
#2、尝试了逐台重启集群所有observer节点 无效
#3、尝试了切rs/手工重新挂在oss目录等 无效
#4、确认select svr_ip, log_archive_status, count(*) from __all_virtual_pg_backup_log_archive_status group by svr_ip, log_archive_status;结果集log_archive_status 都为1
该集群archive log目前 hang住不可操作了

问题：
1、是否有办法让stoping状态的log archive调度彻底停下来，再重开？

#3

不好意思，问题比较多，这个我详细看看，稍等一下

#4

能采集下rs日志吗？如果可以的话，重启下整个集群

#5

信息不够多。可以发一下下面 SQL 结果：

SELECT TENANT_ID , TENANT_NAME , TENANT_TYPE , COMPATIBILITY_MODE ,status, IN_RECYCLEBIN , TENANT_ROLE , LOG_MODE  FROM DBA_OB_TENANTS t;


SELECT TENANT_ID ,DEST_ID ,ROUND_ID ,PIECE_ID ,INCARNATION ,DEST_no,STATUS ,COMPRESSION ,INPUT_BYTES_DISPLAY ,OUTPUT_BYTES_DISPLAY ,COMPRESSION_RATIO ,FILE_STATUS ,path 
FROM oceanbase.CDB_OB_ARCHIVELOG_PIECE_FILES ;

SELECT DEST_ID ,ROUND_ID ,INCARNATION ,DEST_NO ,STATUS ,BASE_PIECE_ID ,USED_PIECE_ID ,INPUT_BYTES_DISPLAY ,OUTPUT_BYTES_DISPLAY ,COMPRESSION ,COMPRESSION_RATIO ,COMMENT , `PATH` 
FROM oceanbase.CDB_OB_ARCHIVELOG ;

SELECT DEST_ID ,ROUND_ID ,INCARNATION ,DEST_NO ,STATUS ,BASE_PIECE_ID ,USED_PIECE_ID ,INPUT_BYTES_DISPLAY ,OUTPUT_BYTES_DISPLAY ,COMPRESSION ,COMPRESSION_RATIO ,COMMENT , `PATH` 
FROM oceanbase.CDB_OB_ARCHIVELOG_SUMMARY  ;



select gmt_create , module , event, name1 , value1 , name2, value2, name3, value3 , rs_svr_ip  from `__all_rootservice_event_history` areh  WHERE 1=1  order by gmt_create desc limit 100

加一些过滤条件，过滤掉无关的事件。

业务租户下查询


SELECT jh.JOB_ID ,jh.INCARNATION ,jh.BACKUP_SET_ID ,jh.INITIATOR_TENANT_ID ,jh.EXECUTOR_TENANT_ID , jh.PLUS_ARCHIVELOG , jh.BACKUP_TYPE , jh.JOB_LEVEL , jh.ENCRYPTION_MODE ,jh.START_TIMESTAMP ,jh.END_TIMESTAMP , jh.STATUS , jh.`RESULT` , jh.COMMENT  
, th.task_ID, th.INPUT_BYTES , th.OUTPUT_BYTES , th.`RESULT` , th.`PATH` 
FROM DBA_OB_BACKUP_JOBS jh, DBA_OB_BACKUP_TASKS th
WHERE jh.JOB_ID =th.JOB_ID AND jh.BACKUP_SET_ID  = th.BACKUP_SET_ID
;


SELECT jh.JOB_ID ,jh.INCARNATION ,jh.BACKUP_SET_ID ,jh.INITIATOR_TENANT_ID ,jh.EXECUTOR_TENANT_ID , jh.PLUS_ARCHIVELOG , jh.BACKUP_TYPE , jh.JOB_LEVEL , jh.ENCRYPTION_MODE ,jh.START_TIMESTAMP ,jh.END_TIMESTAMP , jh.STATUS , jh.`RESULT` , jh.COMMENT  
, th.task_ID, th.INPUT_BYTES , th.OUTPUT_BYTES , th.`RESULT` , th.`PATH` 
FROM DBA_OB_BACKUP_JOB_HISTORY jh, DBA_OB_BACKUP_TASK_HISTORY th
WHERE jh.JOB_ID =th.JOB_ID AND jh.BACKUP_SET_ID  = th.BACKUP_SET_ID
;
#7

因为3.1.4版本没有这些视图，不太好提供，rs信息已经私发给热海同学了.

#10

老铁你这个问题解决了么，我的也卡住了

#11

已经解决了，这个是旧版本的一个缺陷，在3.1.5已经修复，
旧版本用如下步骤处理，注意有一定风险.
i、sys租户
start transaction;
delete from __all_backup_log_archive_status_v2;
delete from __all_backup_log_archive_status_history;
delete from __all_backup_piece_files;
delete from __all_tenant_backup_log_archive_status;
delete from __all_tenant_backup_info;
commit;
ii、切换rs zone到新的zone
alter system switch rootservice leader zone=‘xxx’;

iii、清理user tenant info
alter system change tenant user_tenant;
delete from __all_tenant_backup_log_archive_status;
delete from __all_tenant_backup_info;

iv、重启observer，再重新配置archive

1 个赞