ALTER SYSTEM RESTORE 租户报错

OceanBase
,
#1

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】
【 使用版本 】
【问题描述】清晰明确描述问题
【复现路径】问题出现前后相关操作
【问题现象及影响】

【附件】

obclient [oceanbase]>
obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_RESOURCE_POOLS\G;
*************************** 1. row ***************************
RESOURCE_POOL_ID: 1
NAME: sys_pool
TENANT_ID: 1
CREATE_TIME: 2023-05-08 11:29:06.349333
MODIFY_TIME: 2023-05-08 11:29:06.362046
UNIT_COUNT: 1
UNIT_CONFIG_ID: 1
ZONE_LIST: zone1;zone2;zone3
REPLICA_TYPE: FULL
*************************** 2. row ***************************
RESOURCE_POOL_ID: 1001
NAME: ocp_pool
TENANT_ID: 1002
CREATE_TIME: 2023-05-08 11:29:23.489043
MODIFY_TIME: 2023-05-08 11:29:23.672594
UNIT_COUNT: 1
UNIT_CONFIG_ID: 1001
ZONE_LIST: zone1;zone2;zone3
REPLICA_TYPE: FULL
*************************** 3. row ***************************
RESOURCE_POOL_ID: 1013
NAME: DEMO_OMS_POOL
TENANT_ID: 1008
CREATE_TIME: 2023-05-16 16:25:09.782540
MODIFY_TIME: 2023-05-16 16:27:26.806813
UNIT_COUNT: 1
UNIT_CONFIG_ID: 1013
ZONE_LIST: zone1;zone2;zone3
REPLICA_TYPE: FULL
3 rows in set (0.000 sec)

ERROR:
No query specified

obclient [oceanbase]> SET DECRYPTION IDENTIFIED BY ‘Hello123#’;
Query OK, 0 rows affected (0.001 sec)

obclient [oceanbase]> SELECT * FROM CDB_OB_RESTORE_PROGRESS\G
Empty set (0.006 sec)

obclient [oceanbase]> SELECT * FROM CDB_OB_RESTORE_HISTORY\G
Empty set (0.004 sec)

obclient [oceanbase]>
obclient [oceanbase]>
obclient [oceanbase]>
obclient [oceanbase]> ALTER SYSTEM RESTORE DEMO_OMS_MYSQL_TENANT FROM ‘file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT,file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT’ WITH ‘pool_list=DEMO_OMS_POOL’;

obclient [oceanbase]> SELECT * FROM CDB_OB_RESTORE_HISTORY\G
*************************** 1. row ***************************
TENANT_ID: 1
JOB_ID: 7
RESTORE_TENANT_NAME: DEMO_OMS_MYSQL_TENANT
RESTORE_TENANT_ID: 0
BACKUP_TENANT_NAME: DEMO_OMS_MYSQL_TENANT
BACKUP_TENANT_ID: 1008
BACKUP_CLUSTER_NAME: obcluster
BACKUP_DEST: file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT,file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT
RESTORE_SCN: 1684461049592943425
RESTORE_SCN_DISPLAY: 2023-05-19 09:50:49.592943
RESTORE_OPTION: pool_list=DEMO_OMS_POOL
START_TIMESTAMP: 2023-05-19 10:01:31.587596
FINISH_TIMESTAMP: 2023-05-19 10:02:19.235547
STATUS: FAIL
BACKUP_PIECE_LIST: file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT/piece_d1005r3p3
BACKUP_SET_LIST: file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT/backup_set_7_full
BACKUP_CLUSTER_VERSION: 17179934720
LS_COUNT: 0
FINISH_LS_COUNT: 0
TABLET_COUNT: 0
FINISH_TABLET_COUNT: 0
TOTAL_BYTES: 0
TOTAL_BYTES_DISPLAY: 0.00MB
FINISH_BYTES: 0
FINISH_BYTES_DISPLAY: 0.00MB
DESCRIPTION: NULL
COMMENT: ROOTSERVICE : OB_RESOURCE_POOL_ALREADY_GRANTED(-4626) on “172.22.9.21:2882” with traceid YB42AC160915-0005FBC6D3865DFD-0-0
1 row in set (0.003 sec)

obclient [oceanbase]>
obclient [oceanbase]>
obclient [oceanbase]> SELECT * FROM CDB_OB_RESTORE_PROGRESS\G
Empty set (0.005 sec)

#3

恢复报错。因为资源有限，只能使用原来的资源池

#5

语句中：

  • uri 需要分别指定数据备份路径和日志归档路径；UNTIL [TIME|SCN] 指用户指定的恢复终点，恢复到该位点为止，且包括该位点。
  • 当指定恢复到 TIME 或 SCN 时, 必须以 = 来连接指定的值。
  • restore_option 支持指定 pool_list、locality、primary_zone、kms_encrypt, 其中 pool_list 为必选项, 其余为可选项。

这个不是必备参数吗

#6

obclient [oceanbase]>
obclient [oceanbase]>
obclient [oceanbase]>
obclient [oceanbase]>
obclient [oceanbase]>
obclient [oceanbase]> ALTER SYSTEM RESTORE DEMO_OMS_MYSQL_TENANT FROM ‘file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT,file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT’;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use near '‘file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT,file:///backup/data/arch’ at line 1
obclient [oceanbase]>

去掉语法都不行

#7

在 172.22.9.21 上 grep “YB42AC160915-0005FBC6D3865DFD-0-0” observer.log.20230519*
发一下过滤后的日志附件看看

#8

批量更新税率第二批.txt (125.8 KB)

#9

observer.7z (3.5 MB)

这个是完整的日志

#10

感觉是发生了化身

#11

上面发的2个附件均没有 “YB42AC160915-0005FBC6D3865DFD-0-0” traceid 关键字，请重新发一下。

#12

我找了。

image
image1881×830 79.7 KB

这三个节点都没有。
image
image1881×830 67.5 KB

image
image1881×830 96.4 KB

是不是关键字搞错了

#13

日志一直刷新，需要查看当时报错时候的日志，比如：

grep “YB42AC160915-0005FBC6D3865DFD-0-0” observer.log.20230519*

发一下过滤后的日志附件看看

#14

批量更新税率第二批.txt (14.5 KB)

#15

obclient [oceanbase]>
obclient [oceanbase]>
obclient [oceanbase]> SELECT * FROM CDB_OB_RESTORE_HISTORY\G
*************************** 1. row ***************************
TENANT_ID: 1
JOB_ID: 7
RESTORE_TENANT_NAME: DEMO_OMS_MYSQL_TENANT
RESTORE_TENANT_ID: 0
BACKUP_TENANT_NAME: DEMO_OMS_MYSQL_TENANT
BACKUP_TENANT_ID: 1008
BACKUP_CLUSTER_NAME: obcluster
BACKUP_DEST: file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT,file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT
RESTORE_SCN: 1684461049592943425
RESTORE_SCN_DISPLAY: 2023-05-19 09:50:49.592943
RESTORE_OPTION: pool_list=DEMO_OMS_POOL
START_TIMESTAMP: 2023-05-19 10:01:31.587596
FINISH_TIMESTAMP: 2023-05-19 10:02:19.235547
STATUS: FAIL
BACKUP_PIECE_LIST: file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT/piece_d1005r3p3
BACKUP_SET_LIST: file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT/backup_set_7_full
BACKUP_CLUSTER_VERSION: 17179934720
LS_COUNT: 0
FINISH_LS_COUNT: 0
TABLET_COUNT: 0
FINISH_TABLET_COUNT: 0
TOTAL_BYTES: 0
TOTAL_BYTES_DISPLAY: 0.00MB
FINISH_BYTES: 0
FINISH_BYTES_DISPLAY: 0.00MB
DESCRIPTION: NULL
COMMENT: ROOTSERVICE : OB_RESOURCE_POOL_ALREADY_GRANTED(-4626) on “172.22.9.21:2882” with traceid YB42AC160915-0005FBC6D3865DFD-0-0
*************************** 2. row ***************************
TENANT_ID: 1
JOB_ID: 9
RESTORE_TENANT_NAME: DEMO_OMS_MYSQL_TENANT
RESTORE_TENANT_ID: 0
BACKUP_TENANT_NAME: DEMO_OMS_MYSQL_TENANT
BACKUP_TENANT_ID: 1008
BACKUP_CLUSTER_NAME: obcluster
BACKUP_DEST: file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT,file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT
RESTORE_SCN: 1684461049592943425
RESTORE_SCN_DISPLAY: 2023-05-19 09:50:49.592943
RESTORE_OPTION: pool_list=DEMO_OMS_POOL
START_TIMESTAMP: 2023-05-19 10:15:33.401563
FINISH_TIMESTAMP: 2023-05-19 10:16:19.456947
STATUS: FAIL
BACKUP_PIECE_LIST: file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT/piece_d1005r3p3
BACKUP_SET_LIST: file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT/backup_set_7_full
BACKUP_CLUSTER_VERSION: 17179934720
LS_COUNT: 0
FINISH_LS_COUNT: 0
TABLET_COUNT: 0
FINISH_TABLET_COUNT: 0
TOTAL_BYTES: 0
TOTAL_BYTES_DISPLAY: 0.00MB
FINISH_BYTES: 0
FINISH_BYTES_DISPLAY: 0.00MB
DESCRIPTION: NULL
COMMENT: ROOTSERVICE : OB_RESOURCE_POOL_ALREADY_GRANTED(-4626) on “172.22.9.21:2882” with traceid YB42AC160915-0005FBC6D3865E0C-0-0
*************************** 3. row ***************************
TENANT_ID: 1
JOB_ID: 10
RESTORE_TENANT_NAME: DEMO_OMS_MYSQL_TENANT
RESTORE_TENANT_ID: 0
BACKUP_TENANT_NAME: DEMO_OMS_MYSQL_TENANT
BACKUP_TENANT_ID: 1008
BACKUP_CLUSTER_NAME: obcluster
BACKUP_DEST: file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT,file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT
RESTORE_SCN: 1684461049592943425
RESTORE_SCN_DISPLAY: 2023-05-19 09:50:49.592943
RESTORE_OPTION: pool_list=DEMO_OMS_POOL
START_TIMESTAMP: 2023-05-19 11:05:40.471411
FINISH_TIMESTAMP: 2023-05-19 11:06:20.127637
STATUS: FAIL
BACKUP_PIECE_LIST: file:///backup/data/archive_DEMO_OMS_MYSQL_TENANT/piece_d1005r3p3
BACKUP_SET_LIST: file:///backup/data/database_bak_DEMO_OMS_MYSQL_TENANT/backup_set_7_full
BACKUP_CLUSTER_VERSION: 17179934720
LS_COUNT: 0
FINISH_LS_COUNT: 0
TABLET_COUNT: 0
FINISH_TABLET_COUNT: 0
TOTAL_BYTES: 0
TOTAL_BYTES_DISPLAY: 0.00MB
FINISH_BYTES: 0
FINISH_BYTES_DISPLAY: 0.00MB
DESCRIPTION: NULL
COMMENT: ROOTSERVICE : OB_RESOURCE_POOL_ALREADY_GRANTED(-4626) on “172.22.9.21:2882” with traceid YB42AC160915-0005FBC6D3865E3F-0-0
3 rows in set (0.003 sec)

#16

image
image1881×830 54.2 KB

#17

[2023-05-19 11:06:20.122390] WDIAG [RPC.OBRPC] rpc_call (ob_rpc_proxy.ipp:453) [85428][T1_REST_SER][T1][YB42AC160915-0005FBC6D3865E3F-0-0] [lt=15][errcode=-4626] execute rpc fail(ret=-4626, dst=“172.22.9.21:2882”)
[2023-05-19 11:06:20.122407] WDIAG log_user_error_and_warn (ob_rpc_proxy.cpp:252) [85428][T1_REST_SER][T1][YB42AC160915-0005FBC6D3865E3F-0-0] [lt=17][errcode=-4626] resource pool ‘DEMO_OMS_POOL’ has already been granted to a tenant

看日志是现在 DEMO_OMS_POOL 已经关联了一个租户，一个resource pool 只能绑定到一个租户上。
可以这样确认一下：
select name,tenant_id from dba_ob_resource_pools where name=‘DEMO_OMS_POOL’;

#18

obclient [oceanbase]> select name,tenant_id from dba_ob_resource_pools where name=‘DEMO_OMS_POOL’;
±--------------±----------+
| name | tenant_id |
±--------------±----------+
| DEMO_OMS_POOL | 1008 |
±--------------±----------+
1 row in set (0.002 sec)

#19

obclient [oceanbase]>
obclient [oceanbase]> select name,tenant_id from dba_ob_resource_pools where name=‘DEMO_OMS_POOL’;
±--------------±----------+
| name | tenant_id |
±--------------±----------+
| DEMO_OMS_POOL | 1008 |
±--------------±----------+
1 row in set (0.002 sec)

obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_RESOURCE_POOLS;
±-----------------±--------------±----------±---------------------------±---------------------------±-----------±---------------±------------------±-------------+
| RESOURCE_POOL_ID | NAME | TENANT_ID | CREATE_TIME | MODIFY_TIME | UNIT_COUNT | UNIT_CONFIG_ID | ZONE_LIST | REPLICA_TYPE |
±-----------------±--------------±----------±---------------------------±---------------------------±-----------±---------------±------------------±-------------+
| 1 | sys_pool | 1 | 2023-05-08 11:29:06.349333 | 2023-05-08 11:29:06.362046 | 1 | 1 | zone1;zone2;zone3 | FULL |
| 1001 | ocp_pool | 1002 | 2023-05-08 11:29:23.489043 | 2023-05-08 11:29:23.672594 | 1 | 1001 | zone1;zone2;zone3 | FULL |
| 1013 | DEMO_OMS_POOL | 1008 | 2023-05-16 16:25:09.782540 | 2023-05-16 16:27:26.806813 | 1 | 1013 | zone1;zone2;zone3 | FULL |
±-----------------±--------------±----------±---------------------------±---------------------------±-----------±---------------±------------------±-------------+
3 rows in set (0.000 sec)

#20

DEMO_OMS_POOL 关联哪个租户可以知道吗。我应该就是三个租户啊

#21

批量更新税率第二批.txt (8.9 KB)

#22

可以参考一下物理备份恢复的前置步骤，该resource_pool不能关联对应的租户。
https://www.oceanbase.com/docs/common-oceanbase-database-cn-10000000001692899

也不需要提前创建要恢复的租户，restore的时候会自动创建。