ocp社区版 删除租户副本 任务一直Wait alter locality 一直是执行中

OceanBase 技术问题
,
#1

【 使用环境 】测试环境
【 OB or 其他组件 】OB
【 使用版本 】v4.4.0
【问题描述】ocp社区版 删除租户副本 任务 Wait alter locality 一直是执行中，等了有一个小时了，查询任务表进度是0，不知道卡在哪里了？

image
image2603×1213 354 KB 

obclient(root@sys)[oceanbase]> SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
+-----------+--------+-----------------------+------------+----------+
| TENANT_ID | JOB_ID | JOB_TYPE              | JOB_STATUS | PROGRESS |
+-----------+--------+-----------------------+------------+----------+
|      1006 |      3 | ALTER_TENANT_LOCALITY | INPROGRESS |        0 |
+-----------+--------+-----------------------+------------+----------+
1 row in set (0.010 sec)

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！

2 个赞
#3

查询一下这个

select gmt_create,gmt_modified,job_type,tenant_id,tenant_name,sql_text from __all_rootservice_job where job_status='INPROGRESS' ;
1 个赞
#4

具体操作是一个一个删的么 还是同时执行了多个？

1 个赞
#5 
obclient(root@sys)[oceanbase]> select gmt_create,gmt_modified,job_type,tenant_id,tenant_name,sql_text from __all_rootservice_job where job_status='INPROGRESS';
+----------------------------+----------------------------+-----------------------+-----------+--------------+-----------------------------------------------------------------------+
| gmt_create                 | gmt_modified               | job_type              | tenant_id | tenant_name  | sql_text                                                              |
+----------------------------+----------------------------+-----------------------+-----------+--------------+-----------------------------------------------------------------------+
| 2026-01-20 17:01:28.331872 | 2026-01-20 17:01:28.331872 | ALTER_TENANT_LOCALITY |      1006 | tnt_20260116 | ALTER TENANT `tnt_20260116` LOCALITY = 'FULL{1}@zone1, FULL{1}@zone3' |
+----------------------------+----------------------------+-----------------------+-----------+--------------+-----------------------------------------------------------------------+
1 row in set (0.389 sec)

obclient(root@sys)[oceanbase]>
1 个赞
#6

一次删除了一个副本。

image
image3035×1391 356 KB

1 个赞
#7

一晚上了，现在查询删除进度还是0

obclient(root@sys)[oceanbase]> SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
+-----------+--------+-----------------------+------------+----------+
| TENANT_ID | JOB_ID | JOB_TYPE              | JOB_STATUS | PROGRESS |
+-----------+--------+-----------------------+------------+----------+
|      1006 |      3 | ALTER_TENANT_LOCALITY | INPROGRESS |        0 |
+-----------+--------+-----------------------+------------+----------+
1 row in set (0.007 sec)

obclient(root@sys)[oceanbase]>

`

删除任务执行失败

image
image2586×1358 366 KB

2026-01-20 18:47:20.863 WARN 14538 — [manual-subtask-executor3,a08b38283ed625dc,9e3a8933472c0e8e] c.o.o.c.l.pattern.InterruptibleRetryer : Current thread is interrupted.

2026-01-20 18:47:20.882 ERROR 14538 — [manual-subtask-executor3,a08b38283ed625dc,9e3a8933472c0e8e] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.ob.tenant.alter.locality.not.finished, and args [tnt_20260116]

2026-01-20 18:47:20.898 ERROR 14538 — [manual-subtask-executor3,a08b38283ed625dc,9e3a8933472c0e8e] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : The locality modification of the tnt_20260116 tenant is not completed.

com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=OB_TENANT_ALTER_LOCALITY_NOT_FINISHED, args=tnt_20260116
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96)
at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90)
at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77)
at com.oceanbase.ocp.obops.internal.tenant.task.WaitAlterLocalityTask.checkAlterProgress(WaitAlterLocalityTask.java:40)
at com.oceanbase.ocp.obops.internal.tenant.task.AbstractWaitAlterTenantTask.run(AbstractWaitAlterTenantTask.java:46)
at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35)
at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26)
at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:212)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:206)
at com.oceanbase.ocp.core.ta
sk.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

2026-01-20 18:47:20.849 WARN 14538 — [subtask-executor30,1b66c4128198f1c5,d0c243deeea28acf] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Receive timeout callback, id=20000391, name=Wait alter locality, elapsed=3600, timeout=3600

`

1 个赞
#8

路过学习了

#9

跑一下集群全量巡检，发一下巡检结果

obdiag check run

https://www.oceanbase.com/docs/common-obdiag-cn-1000000005021591

#10

看看这个检查行吗？

obdiag_check_report_observer_2026-01-21-10-23-40.zip (6.0 KB)

#11

巡检结果有一些critical的风险项，应该和这个问题关系不大，我继续看下这个问题

#12

学习学习

#13

使用obdiag收集下最近5分钟的日志发下

#14

日志文件我上传到网盘了，帮忙看一下。

通过网盘分享的文件：observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip
链接: 百度网盘 请输入提取码 提取码: 54jw

#15

网盘我们访问限制了，压缩后多大呢？可以分批上传下

#16

这个得看详细日志了

#18

压缩后有76M，邮件能收不，我发你邮件。

#19

observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip.7z (9.1 MB)

observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip (2).7z (9.0 MB)
observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip (3).7z (9.1 MB)
observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip (4).7z (9.1 MB)
observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip (5).7z (9.0 MB)
observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip (6).7z (9.1 MB)
observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip (7).7z (9.0 MB)
observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip (8).7z (9.0 MB)
observer_log_remote_192_168_56_151_20260122082000_20260122082500_7fe23b.tar.zip (9).7z (4.7 MB)

由于上传限制，我分卷压缩后，又用7z压缩一下，请注意解压顺序。