租户增加副本由4个zone 到 5个zone报错ERROR 1210 (HY000): Incorrect arguments to locality, zone name illegal

[chales675]

【 使用环境 】POC测试环境
【 OB or 其他组件 】 OB
【 使用版本 】ob4.3.5.1
【问题描述】
1.租户增加复本由4个副本调整到5个副本报错如下:
obclient(root@(none))[oceanbase]> ALTER TENANT bk_test locality=“FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3，FULL{1}@zone4, FULL{1}@zone5”;
ERROR 1210 (HY000): Incorrect arguments to locality, zone name illegal
obclient(root@(none))[oceanbase]> select last_trace_id();
±----------------------------------+
| last_trace_id() |
±----------------------------------+
| YB420A100B02-00063352052FCD88-0-0 |
±----------------------------------+
1 row in set (0.003 sec)

2.相关信息：

3.备注：
原集群 3个zone 使用ocp 扩展了zone4 ,zone5 使用ocp 在zone4上添加了副本正常，手工执行ALTER TENANT bk_test locality=“FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3，FULL{1}@zone4, FULL{1}@zone5”; 报错如上。

4.traceid:YB420A100B02-00063352052FCD88-0-0 报错：
observer.log.20250716102102829:[2025-07-16 10:18:58.737560] INFO [SHARE.SCHEMA] serialize_ (ob_schema_struct.cpp:2156) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=23] serialize schema(tenant_id=18446744073709551615, schema_version=1, tenant_name=bk_test, primary_zone=, locked=false, comment=, charset_type=0, collation_type=0, name_case_mode=-1, locality_str=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3，FULL{1}@zone4, FULL{1}@zone5, primary_zone_array=[], default_tablegroup_id=18446744073709551615, default_tablegroup_name=, drop_tenant_time=-1, in_recyclebin=false, arbitration_service_status={status:3}, ret=0)
observer.log.20250716102102829:[2025-07-16 10:18:58.748254] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=69][errcode=-4002] execute rpc fail(addr=“10.16.11.2:2882”, pcode=528, ret=-4002, timeout=999999653)
observer.log.20250716102102829:[2025-07-16 10:18:58.748320] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=53][errcode=-4002] Incorrect arguments to locality, zone name illegal
observer.log.20250716102102829:[2025-07-16 10:18:58.748351] WDIAG [SQL.ENG] execute (ob_tenant_executor.cpp:728) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=17][errcode=-4002] rpc proxy modify tenant failed(ret=-4002)
observer.log.20250716102102829:[2025-07-16 10:18:58.748513] INFO [SHARE] add_event (ob_event_history_table_operator.h:266) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=26] event table add task(ret=0, event_table_name="_all_server_event_history", sql=INSERT INTO all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1752632338748385), ‘sql’, ‘execute_cmd’, ‘cmd_type’, 11, ‘sql_text’, X’414C5445522054454E414E5420626B5F74657374206C6F63616C6974793D2246554C4C7B317D407A6F6E65312C2046554C4C7B317D407A6F6E65322C2046554C4C7B317D407A6F6E6533EFBC8C46554C4C7B317D407A6F6E65342C2046554C4C7B317D407A6F6E653522’, ‘return_code’, -4002, ‘tenant_id’, 1, ‘’, ‘’, ‘xx.xx.xx.2’, 2882))
observer.log.20250716102102829:[2025-07-16 10:18:58.748568] WDIAG [SQL] open_cmd (ob_result_set.cpp:68) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=43][errcode=-4002] execute cmd failed(ret=-4002)
observer.log.20250716102102829:[2025-07-16 10:18:58.748599] WDIAG [SQL] open (ob_result_set.cpp:127) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=26][errcode=-4002] execute plan failed(ret=-4002)
observer.log.20250716102102829:[2025-07-16 10:18:58.748619] WDIAG [SERVER] response_result (ob_sync_cmd_driver.cpp:139) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=13][errcode=-4002] close result set fail(cret=-4002)
observer.log.20250716102102829:[2025-07-16 10:18:58.748649] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=24][errcode=-4002] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err:-4002, err:“OB_INVALID_ARGUMENT”, retry_type:0, client_ret:-4002}, need_retry=false)
observer.log.20250716102102829:[2025-07-16 10:18:58.748693] WDIAG [SERVER] response_result (ob_sync_cmd_driver.cpp:145) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=37][errcode=-4002] result set open failed, check if need retry(ret=-4002, cli_ret=-4002, retry_ctrl.need_retry()=0)
@
【复现路径】问题出现前后相关操作
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接（右键跳转查看）：

【SOP系列 22 】——故障诊断第一步(自助诊断和诊断信息收集)

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！

[论坛小助手]

根据您的描述，问题似乎出现在调整租户 bk_test 副本数的过程中，遇到了错误信息 ERROR 1210 (HY000): Incorrect arguments to locality, zone name illegal，错误代码为 4002。这通常表示在设置副本分布策略时，指定了不合法的 Zone 名称。

根因分析场景: 事务其他错误

根因分析命令: 由于这个问题更倾向于配置问题，而不是事务处理过程中的错误，因此不特别推荐使用 obdiag 的根因分析功能。不过，如果您希望了解是否存在底层的事务错误或其它异常，可以考虑使用以下命令来获取更详细的信息：

obdiag rca run --scene=transaction_other_error

日志采集场景: 单元迁移/缩小 副本不均衡问题

日志采集命令:

obdiag gather scene run --scene=observer.unit_data_imbalance

以上命令会帮助我们收集有关副本迁移或调整的相关日志，以便进一步分析。

问题收集

为了能够更准确地定位和解决问题，请您提供以下信息：

1. 您尝试增加副本数时所使用的具体命令是什么？
2. 当前集群中所有的 Zone 名称是什么？请确保这些 Zone 名称与您在命令中指定的一致。
3. 租户 bk_test 当前的副本分布策略是什么？可以通过 SHOW TENANT 命令查看。
4. 集群的版本信息是什么？这有助于判断是否存在问题已知的 bug 或者是配置上的差异。
5. 如果可能的话，请提供操作前后的一些日志信息，特别是与副本迁移相关的部分。

通过以上信息，我们可以更好地帮助您解决遇到的问题。感谢您的配合！

附上敏捷诊断工具 obdiag 使用帮助链接

[辞霜]

看日志存在RPC报错，查询下是不是zone5的机器与RS的NTP时钟相差是否大于2s

[BrianLiu]

@chales675

ALTER TENANT bk_test locality=“FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3，FULL{1}@zone4, FULL{1}@zone5”;

老师看起来 三副本扩容到无副本。在zone的扩容基础上没问题的情况下，直接ALTER TENANT bk_test locality=“FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3，FULL{1}@zone4, FULL{1}@zone5”; 修改这个locality 理论上是没问题的吧

[chales675]

老师您好：是下面的rpc timeout吗？时间差不是很大 您这看看可能的原因。
observer.log.20250716102102829:[2025-07-16 10:18:58.748254] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [9489][T1_L0_G0][T1][YB420A100B02-00063352052FCD88-0-0] [lt=69][errcode=-4002] execute rpc fail(addr=“xx.xx.xx.2:2882”, pcode=528, ret=-4002, timeout=999999653)

1.rootservice 机器时间（xx.xx.xx.2:2882）
[root@obdemo1 ~]# timedatectl
Local time: Wed 2025-07-16 11:38:10 CST
Universal time: Wed 2025-07-16 03:38:10 UTC
RTC time: Wed 2025-07-16 03:38:10
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a

2. zone5 机器的时间 （xx.xx.xx.6:2882）
   [root@localhost ~]# timedatectl
   Local time: Wed 2025-07-16 11:38:11 CST
   Universal time: Wed 2025-07-16 03:38:11 UTC
   RTC time: Wed 2025-07-16 03:38:11
   Time zone: Asia/Shanghai (CST, +0800)
   NTP enabled: yes
   NTP synchronized: yes
   RTC in local TZ: no
   DST active: n/a

[chales675]

嗯，是有点怪

[辞霜]

看到过类似的问题，原因是zone_name的大小写不对。
可以看下ocp上的zone name和黑屏化查询的一致么

[chales675]

老师您好，看着一样
obclient(root@(none))[oceanbase]> select * from dba_ob_zones\G
*************************** 1. row ***************************
ZONE: zone1
CREATE_TIME: 2025-04-18 17:15:06.961108
MODIFY_TIME: 2025-04-18 17:15:21.371653
STATUS: ACTIVE
IDC: bj
REGION: south
TYPE: ReadWrite
*************************** 2. row ***************************
ZONE: zone2
CREATE_TIME: 2025-04-18 17:15:06.961108
MODIFY_TIME: 2025-04-18 17:15:21.405209
STATUS: ACTIVE
IDC: bj
REGION: south
TYPE: ReadWrite
*************************** 3. row ***************************
ZONE: zone3
CREATE_TIME: 2025-04-18 17:15:06.962171
MODIFY_TIME: 2025-04-18 17:15:21.432830
STATUS: ACTIVE
IDC: bj
REGION: south
TYPE: ReadWrite
*************************** 4. row ***************************
ZONE: zone4
CREATE_TIME: 2025-07-15 10:54:32.824847
MODIFY_TIME: 2025-07-15 10:54:34.787592
STATUS: ACTIVE
IDC: bj
REGION: south
TYPE: ReadWrite
*************************** 5. row ***************************
ZONE: zone5
CREATE_TIME: 2025-07-15 15:06:14.433069
MODIFY_TIME: 2025-07-16 14:27:18.829458
STATUS: ACTIVE
IDC: bj
REGION: south
TYPE: ReadWrite
5 rows in set (0.038 sec)

->

[辞霜]

select * from dba_ob_servers order by zone;再提供一下trace：00063352052FCD88 对应的RS日志。

[和顺]

@chales675

确认一下当前 bk_test 租户在待扩容的 zone5 上是否有资源池

select * from dba_ob_resource_pools 
where tenant_id=(select tenant_id from dba_ob_tenants where tenant_name='bk_test');

如果没有的话，先创建对应的资源池，再调整 locality，具体参考：
https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000003379290

[chales675]

[admin@obdemo1 log]$ grep YB420A100B02-00063352052FCD88-0-0 rootservice*.log
[admin@obdemo1 log]$ grep YB420A100B02-00063352052FCD88-0-0 rootservice*
rootservice.log.20250716113840894:[2025-07-16 10:18:58.737879] INFO [RS] modify_tenant (ob_tenant_ddl_service.cpp:2209) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=38] receive modify tenant request(arg=tenant_schema:{tenant_id:18446744073709551615, schema_version:1, tenant_name:“bk_test”, zone_list:[cnt:0], primary_zone:"", charset_type:0, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:“FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3，FULL{1}@zone4, FULL{1}@zone5”, zone_replica_attr_array:[cnt:0], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:-1, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}, pool_list:[], alter_option_bitset:[512], sys_var_list:[], new_tenant_name:"")
rootservice.log.20250716113840894:[2025-07-16 10:18:58.747749] WDIAG [COMMON] parse_multiple_zone (ob_locality_util.cpp:1306) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=96][errcode=-4002] invalid locality(ret=-4002)
rootservice.log.20250716113840894:[2025-07-16 10:18:58.747830] WDIAG parse_multiple_zone (ob_locality_util.cpp:1307) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=65][errcode=-4002] Incorrect arguments to locality, zone name illegal
rootservice.log.20250716113840894:[2025-07-16 10:18:58.747861] WDIAG [COMMON] parse_locality (ob_locality_util.cpp:1344) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=24][errcode=-4002] fail to parse multiple zone(ret=-4002)
rootservice.log.20250716113840894:[2025-07-16 10:18:58.747877] WDIAG [RS] parse_and_set_create_tenant_new_locality_options (ob_tenant_ddl_service.cpp:4339) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=13][errcode=-4002] fail to parse locality(ret=-4002)
rootservice.log.20250716113840894:[2025-07-16 10:18:58.747903] WDIAG [RS] set_new_tenant_options (ob_tenant_ddl_service.cpp:2341) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=19][errcode=-4002] fail to parse and set new locality option(ret=-4002)
rootservice.log.20250716113840894:[2025-07-16 10:18:58.747929] WDIAG [RS] modify_tenant_inner_phase (ob_tenant_ddl_service.cpp:3600) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=19][errcode=-4002] failed to set new tenant options(ret=-4002)
rootservice.log.20250716113840894:[2025-07-16 10:18:58.747954] WDIAG [RS] modify_tenant (ob_tenant_ddl_service.cpp:2257) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=19][errcode=-4002] modify_tenant_inner_phase fail(ret=-4002)
rootservice.log.20250716113840894:[2025-07-16 10:18:58.747978] WDIAG [RS] modify_tenant (ob_root_service.cpp:2986) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=19][errcode=-4002] ddl service modify tenant failed(arg=tenant_schema:{tenant_id:18446744073709551615, schema_version:1, tenant_name:“bk_test”, zone_list:[cnt:0], primary_zone:"", charset_type:0, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:“FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3，FULL{1}@zone4, FULL{1}@zone5”, zone_replica_attr_array:[cnt:0], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:-1, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}, pool_list:[], alter_option_bitset:[512], sys_var_list:[], new_tenant_name:"", ret=-4002)
rootservice.log.20250716113840894:[2025-07-16 10:18:58.748041] WDIAG [RS] process_ (ob_rs_rpc_processor.h:216) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=54][errcode=-4002] process failed(ret=-4002)
rootservice.log.20250716113840894:[2025-07-16 10:18:58.748062] INFO [RS] process_ (ob_rs_rpc_processor.h:236) [9004][DDLQueueTh0][T0][YB420A100B02-00063352052FCD88-0-0] [lt=15] [DDL] execute ddl like stmt(ret=-4002, cost=10186, ddl_arg=ddl_stmt_str:“ALTER TENANT bk_test locality=“FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3，FULL{1}@zone4, FULL{1}@zone5"”, exec_tenant_id:1, ddl_id_str:”", sync_from_primary:false, based_schema_object_infos:[], parallelism:0, task_id:0, consumer_group_id:0)
[admin@obdemo1 log]$

---

→ select * from dba_ob_servers order by zone\G
*************************** 1. row ***************************
SVR_IP: xx.xx.11.2
SVR_PORT: 2882
ID: 1
ZONE: zone1
SQL_PORT: 2881
WITH_ROOTSERVER: YES
STATUS: ACTIVE
START_SERVICE_TIME: 2025-04-18 17:18:34.030442
STOP_TIME: NULL
BLOCK_MIGRATE_IN_TIME: NULL
CREATE_TIME: 2025-04-18 17:15:07.835410
MODIFY_TIME: 2025-04-18 17:18:36.199150
BUILD_VERSION: 4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)
LAST_OFFLINE_TIME: NULL
*************************** 2. row ***************************
SVR_IP: xx.xx.11.3
SVR_PORT: 2882
ID: 2
ZONE: zone2
SQL_PORT: 2881
WITH_ROOTSERVER: NO
STATUS: ACTIVE
START_SERVICE_TIME: 2025-04-18 17:18:30.448029
STOP_TIME: NULL
BLOCK_MIGRATE_IN_TIME: NULL
CREATE_TIME: 2025-04-18 17:15:07.858034
MODIFY_TIME: 2025-04-18 17:18:32.087609
BUILD_VERSION: 4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)
LAST_OFFLINE_TIME: NULL
*************************** 3. row ***************************
SVR_IP: xx.xx.11.4
SVR_PORT: 2882
ID: 3
ZONE: zone3
SQL_PORT: 2881
WITH_ROOTSERVER: NO
STATUS: ACTIVE
START_SERVICE_TIME: 2025-04-18 17:18:30.459115
STOP_TIME: NULL
BLOCK_MIGRATE_IN_TIME: NULL
CREATE_TIME: 2025-04-18 17:15:07.884599
MODIFY_TIME: 2025-04-18 17:18:32.133342
BUILD_VERSION: 4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)
LAST_OFFLINE_TIME: NULL
*************************** 4. row ***************************
SVR_IP: xx.xx.11.5
SVR_PORT: 2882
ID: 7
ZONE: zone4
SQL_PORT: 2881
WITH_ROOTSERVER: NO
STATUS: ACTIVE
START_SERVICE_TIME: 2025-07-15 11:00:15.897528
STOP_TIME: NULL
BLOCK_MIGRATE_IN_TIME: NULL
CREATE_TIME: 2025-07-15 10:57:33.502389
MODIFY_TIME: 2025-07-15 11:00:17.294710
BUILD_VERSION: 4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)
LAST_OFFLINE_TIME: NULL
*************************** 5. row ***************************
SVR_IP: xx.xx.11.6
SVR_PORT: 2882
ID: 11
ZONE: zone5
SQL_PORT: 2881
WITH_ROOTSERVER: NO
STATUS: ACTIVE
START_SERVICE_TIME: 2025-07-16 14:26:34.797515
STOP_TIME: NULL
BLOCK_MIGRATE_IN_TIME: NULL
CREATE_TIME: 2025-07-15 15:09:17.308884
MODIFY_TIME: 2025-07-16 14:30:03.747809
BUILD_VERSION: 4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)
LAST_OFFLINE_TIME: NULL
5 rows in set (0.008 sec)

obclient(root@(none))[oceanbase]>

(老师注意ip 地址有打码）

[chales675]

和顺老师，这个确实没有

看文档时 "前提条件“ 确实没注意，我试一下稍后反馈

[chales675]

按照老师的建议，该问题已经解决感谢感谢，原因是zone5上没有创建资源池。

复盘一下：
1.将租户bk_test 增加副本由LOCALITY = 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3,调整成 LOCALITY = ‘FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3,FULL{1}@zone4,FULL{1}@zone5’，通过ocp zone4上的副本调整成功，手工增加 zone5 上的副本失败。 原因是手工在zone5上增加副本，应该先在zone5上新增资源池，官方文档有提及但操作过程中没有看仔细。也是自己对原理掌握的不够扎实。

2. 相关建议 ob_locality_util.cpp代码或相关代码如果 能明确是”缺少资源池的问题“那定位问题就简单了。

image1199×430 71.7 KB

3.不太清楚和顺老师如何发现 ”缺少资源池" 的？

4.相关操作记录如下：
CREATE RESOURCE UNIT config_bk_test_zone5_s8_acb
MEMORY_SIZE = ‘8G’,
MAX_CPU = 5, MIN_CPU = 5;

CREATE RESOURCE POOL pool_bk_test_zone5_acb
UNIT=‘config_bk_test_zone5_s8_acb’,
UNIT_NUM=1,
ZONE_LIST=(‘zone5’);

obclient(root@(none))[oceanbase]> ALTER TENANT bk_test RESOURCE_POOL_LIST =(‘pool_bk_test_zone1_lac’,‘pool_bk_test_zone3_znn’,‘pool_bk_test_zone2_cjb’,‘pool_bk_test_zone4_ewb’,‘pool_bk_test_zone5_acb’) ;
Query OK, 0 rows affected (1.143 sec)

obclient(root@(none))[oceanbase]> ALTER TENANT bk_test LOCALITY = ‘FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3,FULL{1}@zone4,FULL{1}@zone5’;
Query OK, 0 rows affected (0.202 sec)

obclient(root@(none))[oceanbase]>

[辞霜]

你的zone4添加是使用的ocp，它会自动帮忙创建资源池。手动添加副本是需要自己来创建资源池的。
可以去看看ocp上任务中心的日志，有完整的流程

[chales675]

感谢老师的支持