执行物理恢复, 一只卡在RESTORING状态

AntTech_3PVK9N · 2026 年4 月 17 日 14:33

【使用环境】测试环境
【 OB or 其他组件】
【使用版本】v4.2.1
【问题描述】使用nfs备份. 执行物理恢复卡在RESTORING状态.

恢复命令:

CREATE RESOURCE UNIT restore_13_50 MAX_CPU 13, MEMORY_SIZE = '50G', MAX_IOPS 10240, MIN_IOPS=10240;
CREATE RESOURCE POOL restore_pool unit = 'restore_13_50', unit_num = 1, zone_list = ('zone1','zone2','zone3');
	
ALTER SYSTEM RESTORE mysql FROM 
  'file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data,
  file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog'
  WITH 'pool_list=restore_pool';

ALTER SYSTEM SET ha_high_thread_score =10 TENANT = mysql;

SELECT * FROM oceanbase.CDB_OB_RESTORE_PROGRESS\G
返回:

*************************** 1. row ***************************
           TENANT_ID: 1
              JOB_ID: 8
 RESTORE_TENANT_NAME: mysql
   RESTORE_TENANT_ID: 1012
  BACKUP_TENANT_NAME: game
    BACKUP_TENANT_ID: 1002
 BACKUP_CLUSTER_NAME: release
         BACKUP_DEST: file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data,file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog
      RESTORE_OPTION: pool_list=restore_pool
         RESTORE_SCN: 1776395583373776496
 RESTORE_SCN_DISPLAY: 2026-04-17 03:13:03.373776
              STATUS: WAIT_TENANT_RESTORE_FINISH
     START_TIMESTAMP: 2026-04-17 11:32:35.226722
     BACKUP_SET_LIST: file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data/backup_set_636_full,file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data/backup_set_637_inc
   BACKUP_PIECE_LIST: file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog/piece_d1009r7p636
         RECOVER_SCN: NULL
 RECOVER_SCN_DISPLAY: NULL
    RECOVER_PROGRESS: NULL
        TABLET_COUNT: NULL
 FINISH_TABLET_COUNT: NULL
    RESTORE_PROGRESS: NULL
         TOTAL_BYTES: NULL
 TOTAL_BYTES_DISPLAY: NULL
        FINISH_BYTES: NULL
FINISH_BYTES_DISPLAY: NULL
         DESCRIPTION: 
*************************** 2. row ***************************
           TENANT_ID: 1012
              JOB_ID: 8
 RESTORE_TENANT_NAME: mysql
   RESTORE_TENANT_ID: 1012
  BACKUP_TENANT_NAME: game
    BACKUP_TENANT_ID: 1002
 BACKUP_CLUSTER_NAME: release
         BACKUP_DEST: file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data,file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog
      RESTORE_OPTION: pool_list=restore_pool
         RESTORE_SCN: 1776395583373776496
 RESTORE_SCN_DISPLAY: 2026-04-17 03:13:03.373776
              STATUS: RESTORING
     START_TIMESTAMP: 2026-04-17 11:32:35.226722
     BACKUP_SET_LIST: file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data/backup_set_636_full,file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data/backup_set_637_inc
   BACKUP_PIECE_LIST: file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog/piece_d1009r7p636
         RECOVER_SCN: 1776395583373776496
 RECOVER_SCN_DISPLAY: 2026-04-17 03:13:03.373776
    RECOVER_PROGRESS: 100.00
        TABLET_COUNT: 39377
 FINISH_TABLET_COUNT: 32456
    RESTORE_PROGRESS: 82.42
         TOTAL_BYTES: 3891674693
 TOTAL_BYTES_DISPLAY: 3.62GB
        FINISH_BYTES: 0
FINISH_BYTES_DISPLAY: 0.00MB
         DESCRIPTION: 
2 rows in set (0.012 sec)

SELECT * FROM oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE module='physical_restore';
返回:

+----------------------------+------------------+-----------------------+----------------+--------+-------------+--------+--------+-------------------------------------------------+-------+--------+-------+--------+-------+--------+------------+---------------+-------------+
| TIMESTAMP                  | MODULE           | EVENT                 | NAME1          | VALUE1 | NAME2       | VALUE2 | NAME3  | VALUE3                                          | NAME4 | VALUE4 | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO | RS_SVR_IP     | RS_SVR_PORT |
+----------------------------+------------------+-----------------------+----------------+--------+-------------+--------+--------+-------------------------------------------------+-------+--------+-------+--------+-------+--------+------------+---------------+-------------+
| 2026-04-17 11:29:25.196264 | physical_restore | restore_start         | ret            | 0      | tenant_name | mysql  |        |                                                 |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:29:25.196287 | physical_restore | change_restore_status | job_id         | 5      | tenant_name | mysql  | status | CREATE_TENANT                                   |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:30:35.597421 | physical_restore | change_restore_status | job_id         | 5      | tenant      | mysql  | status | WAIT_TENANT_RESTORE_FINISH                      |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:30:39.779334 | physical_restore | restore_start         | ret            | -4179  | tenant_name | mysql  |        |                                                 |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:31:23.868675 | physical_restore | restore_start         | ret            | -9014  | tenant_name | mysql  |        |                                                 |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:31:35.607645 | physical_restore | change_restore_status | job_id         | 5      | tenant      | mysql  | status | RESTORE_FAIL                                    |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:31:35.612498 | physical_restore | restore_finish        | restore_status | 8      | tenant      | mysql  |        |                                                 |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:32:35.237599 | physical_restore | restore_start         | ret            | 0      | tenant_name | mysql  |        |                                                 |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:32:35.237620 | physical_restore | change_restore_status | job_id         | 8      | tenant_name | mysql  | status | CREATE_TENANT                                   |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:32:50.955351 | physical_restore | change_restore_status | job_id         | 8      | tenant      | mysql  | status | WAIT_TENANT_RESTORE_FINISH                      |       |        |       |        |       |        |            | 172.31.35.117 |        2882 |
| 2026-04-17 11:33:41.355583 | physical_restore | change_restore_status | job_id         | 8      | tenant      | mysql  | status | RESTORE_CREATE_INIT_LS                          |       |        |       |        |       |        |            | 172.31.45.148 |        2882 |
| 2026-04-17 11:33:41.562746 | physical_restore | change_restore_status | job_id         | 8      | tenant      | mysql  | status | PHYSICAL_RESTORE_WAIT_RESTORE_TO_CONSISTENT_SCN |       |        |       |        |       |        |            | 172.31.45.148 |        2882 |
| 2026-04-17 11:57:41.612975 | physical_restore | change_restore_status | job_id         | 8      | tenant      | mysql  | status | RESTORE_WAIT_LS                                 |       |        |       |        |       |        |            | 172.31.45.148 |        2882 |
+----------------------------+------------------+-----------------------+----------------+--------+-------------+--------+--------+-------------------------------------------------+-------+--------+-------+--------+-------+--------+------------+---------------+-------------+

旭辉 · 2026 年4 月 17 日 14:38

麻烦用obdiag收集下这个问题的信息

obdiag gather scene run --scene=observer.recovery [options]

https://www.oceanbase.com/docs/common-obdiag-cn-1000000005726932

AntTech_3PVK9N · 2026 年4 月 17 日 15:08

感谢您的回复.
已上传, https://zh-cn.tempdrive.zip/s/502d25d1

旭辉 · 2026 年4 月 17 日 16:12

obclient > show variables like 'version_comment';
+-----------------+-------------------------------------------------------------------------------------------------------------------+
| Variable_name   | Value                                                                                                             |
+=================+===================================================================================================================+
| version_comment | OceanBase_CE 4.2.1.11 (r111010012026011510-4f20386337c33093d7caf8fd8098362d78389f10) (Built Jan 15 2026 10:41:13) |
+-----------------+-------------------------------------------------------------------------------------------------------------------+



obclient > SELECT * FROM oceanbase.CDB_OB_RESTORE_PROGRESS limit 20;
+-------------+----------+-----------------------+---------------------+----------------------+--------------------+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------+------------------------+---------------------+----------------------------+----------------------------+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------+---------------------+----------------------------+--------------------+----------------+-----------------------+--------------------+---------------+-----------------------+----------------+------------------------+---------------+
|   TENANT_ID |   JOB_ID | RESTORE_TENANT_NAME   |   RESTORE_TENANT_ID | BACKUP_TENANT_NAME   |   BACKUP_TENANT_ID | BACKUP_CLUSTER_NAME   | BACKUP_DEST                                                                                                                       | RESTORE_OPTION         |         RESTORE_SCN | RESTORE_SCN_DISPLAY        | STATUS                     | START_TIMESTAMP            | BACKUP_SET_LIST                                                                                                                                                          | BACKUP_PIECE_LIST                                                                  |         RECOVER_SCN | RECOVER_SCN_DISPLAY        |   RECOVER_PROGRESS |   TABLET_COUNT |   FINISH_TABLET_COUNT |   RESTORE_PROGRESS |   TOTAL_BYTES | TOTAL_BYTES_DISPLAY   |   FINISH_BYTES | FINISH_BYTES_DISPLAY   | DESCRIPTION   |
+=============+==========+=======================+=====================+======================+====================+=======================+===================================================================================================================================+========================+=====================+============================+============================+============================+==========================================================================================================================================================================+====================================================================================+=====================+============================+====================+================+=======================+====================+===============+=======================+================+========================+===============+
|           1 |        8 | mysql                 |                1012 | game                 |               1002 | release               | file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data,file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog | pool_list=restore_pool | 1776395583373776496 | 2026-04-17 03:13:03.373776 | WAIT_TENANT_RESTORE_FINISH | 2026-04-17 11:32:35.226722 | file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data/backup_set_636_full,file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data/backup_set_637_inc | file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog/piece_d1009r7p636 |                     |                            |                    |                |                       |                    |               |                       |                |                        |               |
+-------------+----------+-----------------------+---------------------+----------------------+--------------------+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------+------------------------+---------------------+----------------------------+----------------------------+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------+---------------------+----------------------------+--------------------+----------------+-----------------------+--------------------+---------------+-----------------------+----------------+------------------------+---------------+
|        1012 |        8 | mysql                 |                1012 | game                 |               1002 | release               | file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data,file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog | pool_list=restore_pool | 1776395583373776496 | 2026-04-17 03:13:03.373776 | RESTORING                  | 2026-04-17 11:32:35.226722 | file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data/backup_set_636_full,file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data/backup_set_637_inc | file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog/piece_d1009r7p636 | 1776395583373776496 | 2026-04-17 03:13:03.373776 |                100 |          39377 |                 32456 |              82.42 |    3891674693 | 3.62GB                |              0 | 0.00MB                 |               |
+-------------+----------+-----------------------+---------------------+----------------------+--------------------+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------+------------------------+---------------------+----------------------------+----------------------------+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------+---------------------+----------------------------+--------------------+----------------+-----------------------+--------------------+---------------+-----------------------+----------------+------------------------+---------------+

旭辉 · 2026 年4 月 17 日 16:27

SELECT * FROM oceanbase.__all_virtual_ls_restore_progress   
WHERE TENANT_ID = 1012;  

SELECT LS_ID, STATUS,   
       FINISH_TABLET_COUNT/TABLET_COUNT AS progress,  
       COMMENT  
FROM oceanbase.__all_virtual_ls_restore_progress   
WHERE TENANT_ID = 1012   
  AND STATUS != 3  -- 非 COMPLETE 状态  
ORDER BY LS_ID;

这两个SQL查询下

旭辉 · 2026 年4 月 17 日 16:28


df -h  
  

free -h  
  

netstat -an | grep ESTABLISHED | wc -l

查询下

AntTech_3PVK9N · 2026 年4 月 17 日 19:14

这两条sql 没有数据返回.

AntTech_3PVK9N · 2026 年4 月 17 日 19:15

AntTech_3PVK9N · 2026 年4 月 17 日 19:23

我发现并不是卡主了, 只是恢复的慢. 从ocp 上看数据盘占用量是一直在增加的. 可这恢复速度也太慢了.
备份大概花了40分钟一共290G数据. 恢复了2个小时. 看数据盘才40个G数据

ALTER SYSTEM SET ha_high_thread_score =100 TENANT = mysql;
将恢复线程设置成了100

sys_bkgd_net_percentage 也设置为了100

恢复的时候cpu 和网络带宽占用是比较低的. 有没有什么办法可以让快点恢复.
按照这个速度估计得一整天呀

Sunshining · 2026 年4 月 17 日 21:02

是不是数据量太大，且磁盘太慢呢

AntTech_3PVK9N · 2026 年4 月 17 日 21:09

不应该, 我备份才用了40分钟左右. 磁盘 500M 吞吐量 8000 ios 利用率很低的.

AntTech_Q5N8RW · 2026 年4 月 17 日 23:16

可以尝试指定在恢复命令里边指定并发度为concurrency=100

ALTER SYSTEM RESTORE mysql FROM
‘file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/data,
file:///nfs_ob/release/1719287282/tenant_incarnation_1/1002/clog’
WITH ‘pool_list=restore_pool&concurrency=100’;

参考：https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000001503172

Tiki · 2026 年4 月 18 日 01:11

帮顶

Amber_www · 2026 年4 月 18 日 09:19

文中提到的restore问题很常见，通过优化tenant和调整RESTORE配置，通常可以解决。

AntTech_JDI52T · 2026 年4 月 18 日 09:26

顶顶

AntTech_3PVK9N · 2026 年4 月 18 日 13:22

非常感谢您的热心回复, 添加concurrency参数可以了

旭辉 · 2026 年4 月 20 日 10:34

配置ha_high_thread_score这个参数时，需要对应的meta租户创建成功后尽快设置效果才明显，
这里更推荐在恢复命令中直接指定 concurrency=100，这样可理解为恢复前参数就已经生效了， pool_list 中的 concurrency 参数是恢复任务的直接配置，优先级高于租户级参数ha_high_thread_score

https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000000218383