命令行修改租户 tnt_20260126 的 Locality， 任务ALTER_TENANT_LOCALITY 一直是执行中。

OceanBase 技术问题
#1

【 使用环境 】测试环境
【 OB or 其他组件 】 OB
【 使用版本 】 4.3.5.5
【问题描述】增加租户副本， 命令行修改租户 tnt_20260126 的 Locality， 任务 ALTER_TENANT_LOCALITY 一直是执行中。

前面有个问题是在ocp删除副本，也是一直卡在 Wait alter locality ，后来删除集群重建再测试删除副本，秒成功。这次测试增加，又卡住了。

ocp社区版 删除租户副本 任务一直Wait alter locality 一直是执行中 - 社区问答- OceanBase社区-分布式数据库

obclient(root@sys)[oceanbase]> ALTER TENANT tnt_20260126 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3";
Query OK, 0 rows affected (0.134 sec)
obclient(root@sys)[oceanbase]>  SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
| JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                         | EXTRA_INFO                                                                              | RS_SVR_IP      | RS_SVR_PORT |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
|      4 | ALTER_TENANT_LOCALITY | SUCCESS    |           0 |      100 | 2026-01-29 09:55:07.663139 | 2026-01-29 09:55:38.787132 |      1002 | ALTER TENANT `tnt_20260126` LOCALITY = 'FULL{1}@zone1, FULL{1}@zone2'            | FROM: 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3', TO: 'FULL{1}@zone1, FULL{1}@zone2' | 192.168.56.151 |        2882 |
|      5 | ALTER_TENANT_LOCALITY | INPROGRESS |        NULL |        0 | 2026-01-29 19:16:34.121401 | 2026-01-29 19:16:34.121401 |      1002 | ALTER TENANT tnt_20260126 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3" | FROM: 'FULL{1}@zone1, FULL{1}@zone2', TO: 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3' | 192.168.56.151 |        2882 |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
2 rows in set (0.011 sec)

obclient(root@sys)[oceanbase]> select * from __all_server_event_history where module like '%storage_ha%' and value1 like '%1002%' and name6 = 'is_failed' and value6 != '0' order by gmt_create desc limit 50;
+----------------------------+----------------+----------+------------+-----------------------------+-----------+--------+-------+--------+-------+-----------------------+-------+-----------------------+---------+-----------------------------------+-----------+--------+------------+
| gmt_create                 | svr_ip         | svr_port | module     | event                       | name1     | value1 | name2 | value2 | name3 | value3                | name4 | value4                | name5   | value5                            | name6     | value6 | extra_info |
+----------------------------+----------------+----------+------------+-----------------------------+-----------+--------+-------+--------+-------+-----------------------+-------+-----------------------+---------+-----------------------------------+-----------+--------+------------+
| 2026-01-29 19:51:04.886794 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5204-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:50:54.823930 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5204-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:50:04.291887 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF51F2-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:49:54.019033 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF51F2-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:49:04.277383 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF51D1-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:48:04.436173 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF51D1-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:47:04.155201 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF51B0-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:46:04.282279 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF51B0-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:46:04.282170 | 192.168.56.153 |     2882 | storage_ha | data_tablets_migration_task | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF51B0-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:45:03.728130 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5189-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:44:04.179132 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5189-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:43:02.878922 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5168-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:42:04.723784 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5168-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:41:02.121032 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5153-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:40:52.618369 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5153-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:40:01.886524 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5132-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:39:06.379731 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5132-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:38:01.475697 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF510E-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:37:04.860768 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF510E-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:36:00.501163 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF50ED-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:35:04.939073 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF50ED-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:34:00.077651 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF50C9-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:33:05.683197 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF50C9-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:31:59.976855 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF50A5-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:31:04.595620 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF50A5-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:29:59.729346 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5081-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:29:03.557320 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5081-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:27:59.569495 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF505D-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:27:04.812512 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF505D-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:25:59.209344 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF504B-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:25:53.066213 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF504B-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:24:58.996890 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5039-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:24:52.166510 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5039-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:23:58.530645 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5027-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:23:51.839895 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5027-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:22:57.698516 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5015-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:22:50.699730 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5015-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:21:57.364012 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5003-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:21:50.253589 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF5003-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:20:57.140745 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF4FF1-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:20:48.352398 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF4FF1-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:19:57.057461 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF4FD0-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:19:05.364406 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF4FD0-0-0 | is_failed | 1      | ADD_LS_OP  |
| 2026-01-29 19:17:57.000997 | 192.168.56.153 |     2882 | storage_ha | ls_ha_finish                | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF4FA9-0-0 | is_failed | -4184  | ADD_LS_OP  |
| 2026-01-29 19:17:04.332283 | 192.168.56.153 |     2882 | storage_ha | migration_finish_task       | tenant_id | 1002   | ls_id | 1      | src   | "192.168.56.153:2882" | dst   | "192.168.56.153:2882" | task_id | YB42C0A83897-0006497CECEF4FA9-0-0 | is_failed | 1      | ADD_LS_OP  |
+----------------------------+----------------+----------+------------+-----------------------------+-----------+--------+-------+--------+-------+-----------------------+-------+-----------------------+---------+-----------------------------------+-----------+--------+------------+
45 rows in set (0.132 sec)

obclient(root@sys)[oceanbase]>
3 个赞
#3

直观能够想到的是时钟是否同步。有没有配置相关的服务！
查看一下192.168.56.153 的observer 日志，是否有相关信息可供参考！

2 个赞
#4 
时间同步没问题。
[root@obs03 log]# chronyc sources -v      
210 Number of sources = 1

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* obd01                        10   6   377    21    +26us[  +26us] +/-  163us
[root@obs03 log]# 

2026-01-29 21:01:15.726447] INFO  [STORAGE] init_for_sstable_replace (ob_tablet.cpp:1058) [127006][T1001_TABLET_MI][T1001][YB42C0A83899-0006497CF9713A80-0-0] [lt=18] succeeded to init tablet for ha build new table store(ret=0, param={tables_handle:{tablet_id:{id:180}, table_count:1, [{i:0, table_key:{tablet_id:{id:180}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"MDS_MINI", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1769403337999178000, v:0}}}, ref:4}]}, rebuild_seq:1, is_transfer_replace:false, start_scn:{val:0, v:0}, tablet_meta:0x7fe59d4bf158, restore_status:0, tablet_split_param:{snapshot_version:-1, multi_version_start:-1, merge_type:0, skip_split_keys:[]}, need_replace_remote_sstable:false, release_mds_scn:{val:1769403337999178000, v:0}}, old_tablet={this:0x7fe5a13e0470, is_inited:true, wash_score:1769691675715460000, hold_ref_cnt:true, ref_cnt:4, version:4, length:0, tablet_addr:{[ver=1,mode=0,seq=42762][2nd=220][3rd=0][offset=405504,size=642,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:180}, data_tablet_id:{id:180}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:1769402237759480003, v:0}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1, multi_version_start:1, compat_mode:0, ha_status:{restore_status:0, data_status:1, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1, cur_report_version:0, data_checksum:0, row_count:0, found_cg_checksum_error:false}, table_store_flag:{with_major_sstable:1, is_user_data_table:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1769403337999178000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, extra_medium_info:{info:4097, compat:1, last_compaction_type:0, wait_check_flag:1, last_medium_scn:1}, last_persisted_committed_tablet_status:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1769402238074347000, v:0}, create_commit_version:1769402238074347000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, start_transfer_commit_version:-1, start_split_commit_version:-1}, create_schema_version:0, space_usage:{all_sstable_data_occupy_size:0, all_sstable_data_required_size:0, all_sstable_meta_size:0, ss_public_sstable_occupy_size:0, tablet_clustered_meta_size:12288, tablet_clustered_sstable_data_size:0, backup_bytes:0}, micro_index_clustered:false, ddl_replay_status:14, split_info:{split_info:0, split_src_tablet_id:{id:0}}, has_truncate_info:false}, table_store_addr:{ptr:0x7fe5a13e0ae0, addr:{[ver=1,mode=0,seq=42742][2nd=424][3rd=0][offset=1277952,size=216,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, storage_schema_addr:{ptr:null, addr:{[ver=1,mode=0,seq=42742][2nd=424][3rd=0][offset=1278168,size=335,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, macro_info_addr:{ptr:0x7fe5a13e0d50, addr:{[ver=1,mode=0,seq=42762][2nd=220][3rd=0][offset=406018,size=128,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}}, mds_data:null, ddl_kvs:null, ddl_kv_count:0, is_external_tablet:false, table_store_cache:{last_major_snapshot_version:1, major_table_cnt:1, minor_table_cnt:0, recycle_version:1, last_major_column_count:12, last_major_macro_block_cnt:0, is_row_store:true, is_tablet_referenced_by_collect_mv:false, last_major_compressor_type:1, last_major_latest_row_store_type:1, last_major_store_type:1}, rowkey_read_info:0x7fe5a13e08d0, is_cs_replica_compat:false, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}, this={this:0x7fe54be96060, is_inited:true, wash_score:1769691675725821000, hold_ref_cnt:true, ref_cnt:1, version:4, length:0, tablet_addr:{[ver=1,mode=0,seq=0][2nd=0][3rd=0][offset=0,size=1120,type=3,seq=0][trans_seq=0, sec_part=0][6th=0]}, allocator:0x7fe59ee55548, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:180}, data_tablet_id:{id:180}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:1769402237759480003, v:0}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1, multi_version_start:1, compat_mode:0, ha_status:{restore_status:0, data_status:1, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1, cur_report_version:0, data_checksum:0, row_count:0, found_cg_checksum_error:false}, table_store_flag:{with_major_sstable:1, is_user_data_table:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1769403337999178000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, extra_medium_info:{info:4097, compat:1, last_compaction_type:0, wait_check_flag:1, last_medium_scn:1}, last_persisted_committed_tablet_status:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1769402238074347000, v:0}, create_commit_version:1769402238074347000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, start_transfer_commit_version:-1, start_split_commit_version:-1}, create_schema_version:0, space_usage:{all_sstable_data_occupy_size:0, all_sstable_data_required_size:0, all_sstable_meta_size:0, ss_public_sstable_occupy_size:0, tablet_clustered_meta_size:298, tablet_clustered_sstable_data_size:2097152, backup_bytes:0}, micro_index_clustered:false, ddl_replay_status:14, split_info:{split_info:0, split_src_tablet_id:{id:0}}, has_truncate_info:false}, table_store_addr:{ptr:0x7fe54be964d0, addr:{[ver=1,mode=0,seq=0][2nd=0][3rd=0][offset=0,size=408,type=3,seq=0][trans_seq=0, sec_part=0][6th=0]}}, storage_schema_addr:{ptr:0x7fe54be97460, addr:{[ver=1,mode=0,seq=0][2nd=0][3rd=0][offset=0,size=408,type=3,seq=0][trans_seq=0, sec_part=0][6th=0]}}, macro_info_addr:{ptr:0x7fe54be97db0, addr:{[ver=1,mode=0,seq=0][2nd=0][3rd=0][offset=0,size=136,type=3,seq=0][trans_seq=0, sec_part=0][6th=0]}}, mds_data:null, ddl_kvs:null, ddl_kv_count:0, is_external_tablet:false, table_store_cache:{last_major_snapshot_version:1, major_table_cnt:1, minor_table_cnt:0, recycle_version:1, last_major_column_count:12, last_major_macro_block_cnt:0, is_row_store:true, is_tablet_referenced_by_collect_mv:false, last_major_compressor_type:1, last_major_latest_row_store_type:1, last_major_store_type:1}, rowkey_read_info:0x7fe54be97b60, is_cs_replica_compat:false, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0})
[2026-01-29 21:01:15.739581] INFO  [COMMON] do_work (ob_tenant_dag_scheduler.cpp:348) [121813][T1001_TABLET_MI][T1001][YB42C0A83899-0006497CF9713A77-0-0] [lt=7] task finish process(ret=0, start_time=1769691675676048, end_time=1769691675739579, runtime=63531, *this={"ObTabletFinishMigrationTask":"ObTabletFinishMigrationTask", this:0x7fe55b540270, ha_dag_net_ctx_:{ObIHADagNetCtx:{"ObIHADagNetCtx":"ObIHADagNetCtx", result_mgr:{result:0, retry_count:1, allow_retry:true, failed_task_id_list:[]}}, arg:{ls_id:{id:1}, type:"ADD_LS_OP", cluster_id:1769339529, priority:0, src:{server:"192.168.56.153:2882", timestamp:1769691660588137, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, dst:{server:"192.168.56.153:2882", timestamp:1769691660588137, flag:2, replica_type:0, region:"default_region", memstore_percent:100}, data_src:{server:"0.0.0.0:0", timestamp:0, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, paxos_replica_number:3, tablet_id_array:[]}, local_clog_checkpoint_scn:{val:1769685394623530002, v:0}, src_ls_rebuild_seq:0, start_ts:1769691661527840, finish_ts:0, task_id:YB42C0A83897-0006497CECEF56DA-0-0, minor_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, major_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, src_ls_meta_package:{ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1769685394623530002, v:0}, clog_base_lsn:{lsn:536838144}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1769416143646892004, v:0}, all_id_meta:{id_meta:[{limited_id:1769691670668862000, latest_log_ts:{val:1769691654042769001, v:0}}, {limited_id:4000001, latest_log_ts:{val:1769664365289358001, v:0}}, {limited_id:70000001, latest_log_ts:{val:1769685394623530002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, palf_meta:{prev_log_info:{log_id:739540, lsn:{lsn:536833459}, scn:{val:1769683636705513000, v:0}, log_proposal_id:2, accum_checksum:1725633896}, curr_lsn:{lsn:536838144}}, dup_ls_meta:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}}, check_tablet_info_cost_time:8035275}, copy_tablet_ctx_:{tablet_id:{id:115}, status:0}, ls_:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:4}, clog_checkpoint_scn:{val:1769686747368133001, v:0}, clog_base_lsn:{lsn:536838144}, rebuild_seq:1, migration_status:1, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1769686640065349003, v:0}, all_id_meta:{id_meta:[{limited_id:1769691690668862000, latest_log_ts:{val:1769691674004254001, v:0}}, {limited_id:4000001, latest_log_ts:{val:1769664365289358001, v:0}}, {limited_id:70000001, latest_log_ts:{val:1769685394623530002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:2, palf_env_:0x7fe5957c2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}})
[2026-01-29 21:01:15.743163] INFO  [STORAGE] build_major_sstable_reuse_info (ob_storage_ha_utils.cpp:1507) [126277][T1001_TABLET_MI][T1001][YB42C0A83899-0006497CF9713A82-0-0] [lt=4] no major sstable, skip build reuse info(ret=0, tablet={this:0x7fe5c640b920, is_inited:true, wash_score:1769691674716055000, hold_ref_cnt:true, ref_cnt:2, version:4, length:0, tablet_addr:{[ver=1,mode=0,seq=42595][2nd=334][3rd=0][offset=311296,size=659,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101096}, data_tablet_id:{id:468}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:1769402237759480003, v:0}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1, multi_version_start:1, compat_mode:0, ha_status:{restore_status:0, data_status:1, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:0, cur_report_version:0, data_checksum:0, row_count:0, found_cg_checksum_error:false}, table_store_flag:{with_major_sstable:1, is_user_data_table:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1769403337999178000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:0}, last_persisted_committed_tablet_status:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1769402238074347000, v:0}, create_commit_version:1769402238074347000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, start_transfer_commit_version:-1, start_split_commit_version:-1}, create_schema_version:0, space_usage:{all_sstable_data_occupy_size:0, all_sstable_data_required_size:0, all_sstable_meta_size:0, ss_public_sstable_occupy_size:0, tablet_clustered_meta_size:8192, tablet_clustered_sstable_data_size:0, backup_bytes:0}, micro_index_clustered:false, ddl_replay_status:14, split_info:{split_info:0, split_src_tablet_id:{id:0}}, has_truncate_info:false}, table_store_addr:{ptr:0x7fe5c640bffc, addr:{[ver=1,mode=0,seq=42594][2nd=321][3rd=0][offset=1216512,size=131,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, storage_schema_addr:{ptr:null, addr:{[ver=1,mode=0,seq=42594][2nd=321][3rd=0][offset=1216643,size=256,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, macro_info_addr:{ptr:0x7fe5c640c194, addr:{[ver=1,mode=0,seq=42595][2nd=334][3rd=0][offset=311827,size=128,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}}, mds_data:null, ddl_kvs:null, ddl_kv_count:0, is_external_tablet:false, table_store_cache:{last_major_snapshot_version:0, major_table_cnt:0, minor_table_cnt:0, recycle_version:0, last_major_column_count:0, last_major_macro_block_cnt:0, is_row_store:true, is_tablet_referenced_by_collect_mv:false, last_major_compressor_type:0, last_major_latest_row_store_type:4, last_major_store_type:0}, rowkey_read_info:0x7fe5c640bd80, is_cs_replica_compat:false, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0})
[2026-01-29 21:01:15.748591] INFO  [COMMON] do_work (ob_tenant_dag_scheduler.cpp:348) [121834][T1001_TABLET_MI][T1001][YB42C0A83899-0006497CF9713A7D-0-0] [lt=6] task finish process(ret=0, start_time=1769691675700418, end_time=1769691675748589, runtime=48171, *this={"ObTabletFinishMigrationTask":"ObTabletFinishMigrationTask", this:0x7fe573d9ef70, ha_dag_net_ctx_:{ObIHADagNetCtx:{"ObIHADagNetCtx":"ObIHADagNetCtx", result_mgr:{result:0, retry_count:1, allow_retry:true, failed_task_id_list:[]}}, arg:{ls_id:{id:1}, type:"ADD_LS_OP", cluster_id:1769339529, priority:0, src:{server:"192.168.56.153:2882", timestamp:1769691660588137, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, dst:{server:"192.168.56.153:2882", timestamp:1769691660588137, flag:2, replica_type:0, region:"default_region", memstore_percent:100}, data_src:{server:"0.0.0.0:0", timestamp:0, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, paxos_replica_number:3, tablet_id_array:[]}, local_clog_checkpoint_scn:{val:1769685394623530002, v:0}, src_ls_rebuild_seq:0, start_ts:1769691661527840, finish_ts:0, task_id:YB42C0A83897-0006497CECEF56DA-0-0, minor_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, major_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, src_ls_meta_package:{ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1769685394623530002, v:0}, clog_base_lsn:{lsn:536838144}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1769416143646892004, v:0}, all_id_meta:{id_meta:[{limited_id:1769691670668862000, latest_log_ts:{val:1769691654042769001, v:0}}, {limited_id:4000001, latest_log_ts:{val:1769664365289358001, v:0}}, {limited_id:70000001, latest_log_ts:{val:1769685394623530002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, palf_meta:{prev_log_info:{log_id:739540, lsn:{lsn:536833459}, scn:{val:1769683636705513000, v:0}, log_proposal_id:2, accum_checksum:1725633896}, curr_lsn:{lsn:536838144}}, dup_ls_meta:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}}, check_tablet_info_cost_time:8035275}, copy_tablet_ctx_:{tablet_id:{id:430}, status:0}, ls_:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:4}, clog_checkpoint_scn:{val:1769686747368133001, v:0}, clog_base_lsn:{lsn:536838144}, rebuild_seq:1, migration_status:1, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1769686640065349003, v:0}, all_id_meta:{id_meta:[{limited_id:1769691690668862000, latest_log_ts:{val:1769691674004254001, v:0}}, {limited_id:4000001, latest_log_ts:{val:1769664365289358001, v:0}}, {limited_id:70000001, latest_log_ts:{val:1769685394623530002, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:2, palf_env_:0x7fe5957c2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}})
2 个赞
#5

把OBserver集群停掉，然后再启动起来，任务 ALTER_TENANT_LOCALITY 还是执行中。而且不停刷日志，能不能把这个任务停掉，或者回滚。

image
image1123×792 39.8 KB 

obclient(root@sys)[oceanbase]>  SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
| JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                         | EXTRA_INFO                                                                              | RS_SVR_IP      | RS_SVR_PORT |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
|      4 | ALTER_TENANT_LOCALITY | SUCCESS    |           0 |      100 | 2026-01-29 09:55:07.663139 | 2026-01-29 09:55:38.787132 |      1002 | ALTER TENANT `tnt_20260126` LOCALITY = 'FULL{1}@zone1, FULL{1}@zone2'            | FROM: 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3', TO: 'FULL{1}@zone1, FULL{1}@zone2' | 192.168.56.151 |        2882 |
|      5 | ALTER_TENANT_LOCALITY | INPROGRESS |        NULL |        0 | 2026-01-29 19:16:34.121401 | 2026-01-29 19:16:34.121401 |      1002 | ALTER TENANT tnt_20260126 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3" | FROM: 'FULL{1}@zone1, FULL{1}@zone2', TO: 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3' | 192.168.56.151 |        2882 |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
2 rows in set (0.003 sec)

[2026-01-30 08:27:28.573578] INFO  [STORAGE] dec_ref_with_aggregated_info (ob_tablet.cpp:3580) [10241][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=3] the tablet that decreases ref cnt is(is_inited_=true, tablet_meta_.ls_id_={id:1}, tablet_meta_.tablet_id_={id:378}, table_store_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=253952,size=131,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, storage_schema_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=254083,size=305,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, tablet_addr_={[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=192512,size=647,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, this=0x7f0f58abaa10, info_iter={macro_info:{entry_block:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, meta_block_info_arr:{cnt:0, arr:null, capacity:0}, data_block_info_arr:{cnt:0, arr:null, capacity:0}, shared_meta_block_info_arr:{cnt:1, arr:0x7f0f58abb2c4, capacity:1}, shared_data_block_info_arr:{cnt:0, arr:null, capacity:0}, is_inited:true}, cur_type:4, target_type:6, is_linked:false}, lbt()="0x263f18e8 0x1d72212c 0x1d721708 0x1d70670a 0x1c98ce6c 0xa7ce35d 0xa7c0843 0xa7bf255 0xa7b13f0 0x2607b9a5 0x26079c32 0x7f0fc7ea5dd5 0x7f0fc7bceead")
[2026-01-30 08:27:28.573588] INFO  [STORAGE] reset (ob_tablet.cpp:243) [10241][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=8] reset tablet(this=0x7f0f58abaa10, ls_id={id:1}, tablet_id={id:378}, lbt()="0x263f18e8 0x1d6d6003 0x1d6d57b3 0x1d6d4d63 0x1c9febb8 0x1c975cda 0x1c98cf23 0xa7ce35d 0xa7c0843 0xa7bf255 0xa7b13f0 0x2607b9a5 0x26079c32 0x7f0fc7ea5dd5 0x7f0fc7bceead")
[2026-01-30 08:27:28.573586] INFO  [STORAGE] inc_ref_with_macro_iter (ob_tablet.cpp:3400) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=2] the tablet that inner increases ref cnt is(ret=0, hold_ref_cnt_=false, ls_id={id:1}, tablet_id={id:60253}, table_store_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=499712,size=131,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, storage_schema_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=499843,size=192,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, tablet_addr_={[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=438272,size=635,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, this=0x7f0f593540a0, macro_iter={macro_info:{entry_block:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, meta_block_info_arr:{cnt:0, arr:null, capacity:0}, data_block_info_arr:{cnt:0, arr:null, capacity:0}, shared_meta_block_info_arr:{cnt:1, arr:0x7f0f5935490c, capacity:1}, shared_data_block_info_arr:{cnt:0, arr:null, capacity:0}, is_inited:true}, cur_type:4, target_type:6, is_linked:false}, lbt()="0x263f18e8 0x1d71fadc 0x1d6bda4b 0x1d6dacff 0x1d6b70b4 0x1d6b60a7 0x1c294b31 0x1c29e563 0x1e0431bb 0x1e05a230 0x106739cf 0x106714ee 0x2607b9a5 0x26079c32 0x7f0fc7ea5dd5 0x7f0fc7bceead")
[2026-01-30 08:27:28.573593] INFO  [STORAGE] gc_tablets_in_queue (ob_tenant_meta_mem_mgr.cpp:893) [10241][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=3] succeed to gc tablet(ret=0, tablet=0x7f0f58abaa10)
[2026-01-30 08:27:28.573598] INFO  [STORAGE] dec_ref_with_aggregated_info (ob_tablet.cpp:3580) [10241][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=2] the tablet that decreases ref cnt is(is_inited_=true, tablet_meta_.ls_id_={id:1}, tablet_meta_.tablet_id_={id:101036}, table_store_addr_.addr_={[ver=1,mode=re_addr:{ptr:0x7f0f58b8832c, addr:{[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1327104,size=131,type=2,seq=1][trans_seq=0, sec_part=0][6th=1]}}, storage_schema_addr:{ptr:null, addr:{[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1327235,size=192,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, macro_info_addr:{ptr:0x7f0f58b884c4, addr:{[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1266171,size=128,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}}, mds_data:null, ddl_kvs:null, ddl_kv_count:0, is_external_tablet:false, table_store_cache:{last_major_snapshot_version:0, major_table_cnt:0, minor_table_cnt:0, recycle_version:0, last_major_column_count:0, last_major_macro_block_cnt:0, is_row_store:true, is_tablet_referenced_by_collect_mv:false, last_major_compressor_type:0, last_major_latest_row_store_type:4, last_major_store_type:0}, rowkey_read_info:0x7f0f58b88140, is_cs_replica_compat:false, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0})
[2026-01-30 08:27:28.589385] INFO  [STORAGE] update_tablet_release_memtable_for_offline (ob_ls_tablet_service.cpp:1386) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=16] succeeded to copy tablet to release memtable(ret=0, key={ls_id:{id:1}, tablet_id:{id:60404}}, tablet_handle={obj:0x7f0f592d4fb0, obj_pool:0x7f0f8d235930, allocator:null, wash_priority:0, allow_copy_and_assign:true}, new_tablet_handle={obj:0x7f0f58b87ce0, obj_pool:0x7f0f8d235930, allocator:null, wash_priority:0, allow_copy_and_assign:true})
[2026-01-30 08:27:28.589391] INFO  [STORAGE] transform (ob_tablet_persister.cpp:1283) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=3] TINY TABLET: tablet + rowkey_read_info + tablet store + auto_inc_seq(buf=0x7f0f58bc5ec0, start_pos=2020, remain=1764)
[2026-01-30 08:27:28.589395] INFO  [STORAGE] inc_ref_with_macro_iter (ob_tablet.cpp:3400) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=2] the tablet that inner increases ref cnt is(ret=0, hold_ref_cnt_=false, ls_id={id:1}, tablet_id={id:60532}, table_store_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1335296,size=131,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, storage_schema_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1335427,size=192,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, tablet_addr_={[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1273856,size=635,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, this=0x7f0f58bc5ec0, macro_iter={macro_info:{entry_block:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, meta_block_info_arr:{cnt:0, arr:null, capacity:0}, data_block_info_arr:{cnt:0, arr:null, capacity:0}, shared_meta_block_info_arr:{cnt:1, arr:0x7f0f58bc672c, capacity:1}, shared_data_block_info_arr:{cnt:0, arr:null, capacity:0}, is_inited:true}, cur_type:4, target_type:6, is_linked:false}, lbt()="0x263f18e8 0x1d71fadc 0x1d6bda4b 0x1d6dacff 0x1d6b70b4 0x1d6b60a7 0x1c294b31 0x1c29e563 0x1e0431bb 0x1e05a230 0x106739cf 0x106714ee 0x2607b9a5 0x26079c32 0x7f0fc7ea5dd5 0x7f0fc7bceead")
[2026-01-30 08:27:28.589431] INFO  [STORAGE] rebuild_memtables (ob_tablet.cpp:5416) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=34] table store update memtable success(this={this:0x7f0f58bc5ec0, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:4, length:0, tablet_addr:{[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1273856,size=635,type=4,seq=1][trans_seq=0, sec_part=0][6th=1]}, allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:60532}, data_tablet_id:{id:532}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:1769402237759480003, v:0}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1, multi_version_start:1, compat_mode:0, ha_status:{restore_status:0, data_status:1, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:0, cur_report_version:0, data_checksum:0, row_count:0, found_cg_checksum_error:false}, table_store_flag:{with_major_sstable:1, is_user_data_table:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1769402607272047000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:0}, last_persisted_committed_tablet_status:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1769402238074347000, v:0}, create_commit_version:1769402238074347000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, start_transfer_commit_version:-1, start_split_commit_version:-1}, create_schema_version:0, space_usage:{all_sstable_data_occupy_size:0, all_sstable_data_required_size:0, all_sstable_meta_size:0, ss_public_sstable_occupy_size:0, tablet_clustered_meta_size:8192, tablet_clustered_sstable_data_size:0, backup_bytes:0}, micro_index_clustered:false, ddl_replay_status:14, split_info:{split_info:0, split_src_tablet_id:{id:0}}, has_truncate_info:false}, table_store_addr:{ptr:0x7f0f58bc650c, addr:{[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1335296,size=131,type=2,seq=1][trans_seq=0, sec_part=0][6th=1]}}, storage_schema_addr:{ptr:null, addr:{[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1335427,size=192,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, macro_info_addr:{ptr:0x7f0f58bc66a4, addr:{[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1274363,size=128,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}}, mds_data:null, ddl_kvs:null, ddl_kv_count:0, is_external_tablet:false, table_store_cache:{last_major_snapshot_version:0, major_table_cnt:0, minor_table_cnt:0, recycle_version:0, last_major_column_count:0, last_major_macro_block_cnt:0, is_row_store:true, is_tablet_referenced_by_collect_mv:false, last_major_compressor_type:0, last_major_latest_row_store_type:4, last_major_store_type:0}, rowkey_read_info:0x7f0f58bc6320, is_cs_replica_compat:false, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0})
[2026-01-30 08:27:28.589463] INFO  [STORAGE] update_tablet_release_memtable_for_offline (ob_ls_tablet_service.cpp:1386) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=31] succeeded to copy tablet to release memtable(ret=0, key={ls_id:{id:1}, tablet_id:{id:60532}}, tablet_handle={obj:0x7f0f592d5ec0, obj_pool:0x7f0f8d235930, allocator:null, wash_priority:0, allow_copy_and_assign:true}, new_tablet_handle={obj:0x7f0f58bc5ec0, obj_pool:0x7f0f8d235930, allocator:null, wash_priority:0, allow_copy_and_assign:true})
[2026-01-30 08:27:28.589473] INFO  [STORAGE] transform (ob_tablet_persister.cpp:1283) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=4] TINY TABLET: tablet + rowkey_read_info + tablet store + auto_inc_seq(buf=0x7f0f58ba5ec0, start_pos=2056, remain=1728)
[2026-01-30 08:27:28.589479] INFO  [STORAGE] inc_ref_with_macro_iter (ob_tablet.cpp:3400) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=3] the tablet that inner increases ref cnt is(ret=0, hold_ref_cnt_=false, ls_id={id:1}, tablet_id={id:50441}, table_store_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1339392,size=131,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, storage_schema_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1339523,size=258,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, tablet_addr_={[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1277952,size=641,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, this=0x7f0f58ba5ec0, macro_iter={macro_info:{entry_block:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, meta_block_info_arr:{cnt:0, arr:null, capacity:0}, data_block_info_arr:{cnt:0, arr:null, capacity:0}, shared_meta_block_info_arr:{cnt:1, arr:0x7f0f58ba6750, capacity:1}, shared_data_block_info_arr:{cnt:0, arr:null, capacity:0}, is_inited:true}, cur_type:4, target_type:6, is_linked:false}, lbt()="0x263f18e8 0x1d71fadc 0x1d6bda4b 0x1d6dacff 0x1d6b70b4 0x1d6b60a7 0x1c294b31 0x1c29e563 0x1e0431bb 0x1e05a230 0x106739cf 0x106714ee 0x2607b9a5 0x26079c32 0x7f0fc7ea5dd5 0x7f0fc7bceead")
[2026-01-30 08:27:28.590653] INFO  [STORAGE] rebuild_memtables (ob_tablet.cpp:5416) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=1162] table store update memtable success(this={this:0x7f0f58ba5ec0, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:4, length:0, tablet_addr:{[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1277952,size=641,type=4,seq=1][trans_seq=0, sec_part=0][6th=1]}, allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:50441}, data_tablet_id:{id:441}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:1769402237759480003, v:0}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1, multi_version_start:1, compat_mode:0, ha_status:{restore_status:0, data_status:1, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:0, cur_report_version:0, data_checksum:0, row_count:0, found_cg_checksum_error:false}, table_store_flag:{with_major_sstable:1, is_user_data_table:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1769402607272047000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:0}, last_persisted_committed_tablet_status:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1769402238074347000, v:0}, create_commit_version:1769402238074347000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, start_transfer_commit_version:-1, start_split_commit_version:-1}, create_schema_version:0, space_usage:{all_sstable_data_occupy_size:0, all_sstable_data_required_size:0, all_sstable_meta_size:0, ss_public_sstable_occupy_size:0, tablet_clustered_meta_size:8192, tablet_clustered_sstable_data_size:0, backup_bytes:0}, micro_index_clustered:false, ddl_replay_status:14, split_info:{split_info:0, split_src_tablet_id:{id:0}}, has_truncate_info:false}, table_store_addr:{ptr:0x7f0f58ba6530, addr:{[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1339392,size=131,type=2,seq=1][trans_seq=0, sec_part=0][6th=1]}}, storage_schema_addr:{ptr:null, addr:{[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1339523,size=258,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, macro_info_addr:{ptr:0x7f0f58ba66c8, addr:{[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1278465,size=128,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}}, mds_data:null, ddl_kvs:null, ddl_kv_count:0, is_external_tablet:false, table_store_cache:{last_major_snapshot_version:0, major_table_cnt:0, minor_table_cnt:0, recycle_version:0, last_major_column_count:0, last_major_macro_block_cnt:0, is_row_store:true, is_tablet_referenced_by_collect_mv:false, last_major_compressor_type:0, last_major_latest_row_store_type:4, last_major_store_type:0}, rowkey_read_info:0x7f0f58ba6320, is_cs_replica_compat:false, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0})
[2026-01-30 08:27:28.590706] INFO  [STORAGE] update_tablet_release_memtable_for_offline (ob_ls_tablet_service.cpp:1386) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=45] succeeded to copy tablet to release memtable(ret=0, key={ls_id:{id:1}, tablet_id:{id:50441}}, tablet_handle={obj:0x7f0f58b35ec0, obj_pool:0x7f0f8d235930, allocator:null, wash_priority:0, allow_copy_and_assign:true}, new_tablet_handle={obj:0x7f0f58ba5ec0, obj_pool:0x7f0f8d235930, allocator:null, wash_priority:0, allow_copy_and_assign:true})
[2026-01-30 08:27:28.590733] INFO  [STORAGE] transform (ob_tablet_persister.cpp:1283) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9] TINY TABLET: tablet + rowkey_read_info + tablet store + auto_inc_seq(buf=0x7f0f8d190a10, start_pos=2056, remain=1728)
[2026-01-30 08:27:28.590950] INFO  [STORAGE] inc_ref_with_macro_iter (ob_tablet.cpp:3400) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] the tablet that inner increases ref cnt is(ret=0, hold_ref_cnt_=false, ls_id={id:1}, tablet_id={id:389}, table_store_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1343488,size=131,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, storage_schema_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1343619,size=583,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, tablet_addr_={[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1282048,size=641,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, this=0x7f0f8d190a10, macro_iter={macro_info:{entry_block:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, meta_block_info_arr:{cnt:0, arr:null, capacity:0}, data_block_info_arr:{cnt:0, arr:null, capacity:0}, shared_meta_block_info_arr:{cnt:1, arr:0x7f0f8d1912a0, capacity:1}, shared_data_block_info_arr:{cnt:0, arr:null, capacity:0}, is_inited:true}, cur_type:4, target_type:6, is_linked:false}, lbt()="0x263f18e8 0x1d71fadc 0x1d6bda4b 0x1d6dacff 0x1d6b70b4 0x1d6b60a7 0x1c294b31 0x1c29e563 0x1e0431bb 0x1e05a230 0x106739cf 0x106714ee 0x2607b9a5 0x26079c32 0x7f0fc7ea5dd5 0x7f0fc7bceead")
[2026-01-30 08:27:28.591013] INFO  [STORAGE] rebuild_memtables (ob_tablet.cpp:5416) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=59] table store update memtable success(this={this:0x7f0f8d190a10, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:4, length:0, tablet_addr:{[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1282048,size=641,type=4,seq=1][trans_seq=0, sec_part=0][6th=1]}, allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:389}, data_tablet_id:{id:389}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:1769402237759480003, v:0}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1, multi_version_start:1, compat_mode:0, ha_status:{restore_status:0, data_status:1, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:0, cur_report_version:0, data_checksum:0, row_count:0, found_cg_checksum_error:false}, table_store_flag:{with_major_sstable:1, is_user_data_table:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1769402607272047000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:0}, last_persisted_committed_tablet_status:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1769402238074347000, v:0}, create_commit_version:1769402238074347000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, start_transfer_commit_version:-1, start_split_commit_version:-1}, create_schema_version:0, space_usage:{all_sstable_data_occupy_size:0, all_sstable_data_required_size:0, all_sstable_meta_size:0, ss_public_sstable_occupy_size:0, tablet_clustered_meta_size:8192, tablet_clustered_sstable_data_size:0, backup_bytes:0}, micro_index_clustered:false, ddl_replay_status:14, split_info:{split_info:0, split_src_tablet_id:{id:0}}, has_truncate_info:false}, table_store_addr:{ptr:0x7f0f8d191080, addr:{[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1343488,size=131,type=2,seq=1][trans_seq=0, sec_part=0][6th=1]}}, storage_schema_addr:{ptr:null, addr:{[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1343619,size=583,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, macro_info_addr:{ptr:0x7f0f8d191218, addr:{[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1282561,size=128,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}}, mds_data:null, ddl_kvs:null, ddl_kv_count:0, is_external_tablet:false, table_store_cache:{last_major_snapshot_version:0, major_table_cnt:0, minor_table_cnt:0, recycle_version:0, last_major_column_count:0, last_major_macro_block_cnt:0, is_row_store:true, is_tablet_referenced_by_collect_mv:false, last_major_compressor_type:0, last_major_latest_row_store_type:4, last_major_store_type:0}, rowkey_read_info:0x7f0f8d190e70, is_cs_replica_compat:false, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0})
[2026-01-30 08:27:28.591046] INFO  [STORAGE] update_tablet_release_memtable_for_offline (ob_ls_tablet_service.cpp:1386) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=30] succeeded to copy tablet to release memtable(ret=0, key={ls_id:{id:1}, tablet_id:{id:389}}, tablet_handle={obj:0x7f0f592d1380, obj_pool:0x7f0f8d235930, allocator:null, wash_priority:0, allow_copy_and_assign:true}, new_tablet_handle={obj:0x7f0f8d190a10, obj_pool:0x7f0f8d235930, allocator:null, wash_priority:0, allow_copy_and_assign:true})
[2026-01-30 08:27:28.591065] INFO  [STORAGE] transform (ob_tablet_persister.cpp:1283) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=6] TINY TABLET: tablet + rowkey_read_info + tablet store + auto_inc_seq(buf=0x7f0f8d0c8290, start_pos=2092, remain=1692)
[2026-01-30 08:27:28.591073] INFO  [STORAGE] inc_ref_with_macro_iter (ob_tablet.cpp:3400) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=4] the tablet that inner increases ref cnt is(ret=0, hold_ref_cnt_=false, ls_id={id:1}, tablet_id={id:101035}, table_store_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1351680,size=131,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, storage_schema_addr_.addr_={[ver=1,mode=0,seq=55669][2nd=295][3rd=0][offset=1351811,size=204,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, tablet_addr_={[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1290240,size=647,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, this=0x7f0f8d0c8290, macro_iter={macro_info:{entry_block:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, meta_block_info_arr:{cnt:0, arr:null, capacity:0}, data_block_info_arr:{cnt:0, arr:null, capacity:0}, shared_meta_block_info_arr:{cnt:1, arr:0x7f0f8d0c8b44, capacity:1}, shared_data_block_info_arr:{cnt:0, arr:null, capacity:0}, is_inited:true}, cur_type:4, target_type:6, is_linked:false}, lbt()="0x263f18e8 0x1d71fadc 0x1d6bda4b 0x1d6dacff 0x1d6b70b4 0x1d6b60a7 0x1c294b31 0x1c29e563 0x1e0431bb 0x1e05a230 0x106739cf 0x106714ee 0x2607b9a5 0x26079c32 0x7f0fc7ea5dd5 0x7f0fc7bceead")
[2026-01-30 08:27:28.591081] INFO  [STORAGE] rebuild_memtables (ob_tablet.cpp:5416) [10262][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=8] table store update memtable success(this={this:0x7f0f8d0c8290, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:4, length:0, tablet_addr:{[ver=1,mode=0,seq=55670][2nd=449][3rd=0][offset=1290240,size=647,type=4,seq=1][trans_seq=0, sec_part=0][6th=1]}, allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101035}^C
[admin@obs03 log]$
2 个赞
#6

这几个信息 查一下
SELECT TENANT_ID, LS_ID, STATUS, PRIMARY_ZONE, UNIT_GROUP_ID, LS_GROUP_ID FROM CDB_OB_LS WHERE TENANT_ID = 1002 AND LS_ID != 1 AND FLAG = ‘’;
SELECT * FROM V$OB_LS_REPLICA_TASK_PLAN WHERE TENANT_ID = 1002;
SELECT TENANT_ID, TENANT_NAME, PRIMARY_ZONE, LOCALITY, PREVIOUS_LOCALITY FROM DBA_OB_TENANTS WHERE TENANT_ID = 1002;
SELECT * FROM CDB_OB_TRANSFER_TASKS WHERE TENANT_ID = 1002;

1 个赞
#7 
obclient(root@sys)[oceanbase]> SELECT TENANT_ID, LS_ID, STATUS, PRIMARY_ZONE, UNIT_GROUP_ID, LS_GROUP_ID FROM CDB_OB_LS WHERE TENANT_ID = 1002 AND LS_ID != 1 AND FLAG = '';
+-----------+-------+--------+-------------------+---------------+-------------+
| TENANT_ID | LS_ID | STATUS | PRIMARY_ZONE      | UNIT_GROUP_ID | LS_GROUP_ID |
+-----------+-------+--------+-------------------+---------------+-------------+
|      1002 |  1001 | NORMAL | zone1;zone2;zone3 |          1001 |        1001 |
|      1002 |  1002 | NORMAL | zone2;zone1;zone3 |          1001 |        1001 |
+-----------+-------+--------+-------------------+---------------+-------------+
2 rows in set (0.031 sec)

obclient(root@sys)[oceanbase]> SELECT * FROM V$OB_LS_REPLICA_TASK_PLAN WHERE TENANT_ID = 1002;
+-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+
| TENANT_ID | LS_ID | TASK_TYPE   | PRIORITY | TARGET_REPLICA_SVR_IP | TARGET_REPLICA_SVR_PORT | TARGET_PAXOS_REPLICA_NUMBER | TARGET_REPLICA_TYPE | SOURCE_REPLICA_SVR_IP | SOURCE_REPLICA_SVR_PORT | SOURCE_PAXOS_REPLICA_NUMBER | SOURCE_REPLICA_TYPE | TASK_EXEC_SVR_IP | TASK_EXEC_SVR_PORT | COMMENT                                 |
+-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+
|      1002 |     1 | ADD REPLICA | HIGH     | 192.168.56.153        |                    2882 |                           3 | FULL                | 192.168.56.151        |                    2882 |                           2 | FULL                | 192.168.56.153   |               2882 | add paxos replica according to locality |
+-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+
1 row in set (0.183 sec)

obclient(root@sys)[oceanbase]> SELECT TENANT_ID, TENANT_NAME, PRIMARY_ZONE, LOCALITY, PREVIOUS_LOCALITY FROM DBA_OB_TENANTS WHERE TENANT_ID = 1002;
+-----------+--------------+-------------------+---------------------------------------------+------------------------------+
| TENANT_ID | TENANT_NAME  | PRIMARY_ZONE      | LOCALITY                                    | PREVIOUS_LOCALITY            |
+-----------+--------------+-------------------+---------------------------------------------+------------------------------+
|      1002 | tnt_20260126 | zone1,zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | FULL{1}@zone1, FULL{1}@zone2 |
+-----------+--------------+-------------------+---------------------------------------------+------------------------------+
1 row in set (0.136 sec)

obclient(root@sys)[oceanbase]> SELECT * FROM CDB_OB_TRANSFER_TASKS WHERE TENANT_ID = 1002;
Empty set (0.137 sec)

obclient(root@sys)[oceanbase]>
1 个赞
#8

根据这个trace_id 查看一下 rootservice.log和observer.log日志信息 麻烦提供一下

select * from GV$OB_SEVERS;这个信息 也查一下

1 个赞
#9 
日志产生的太多了，前面删除了，observer.log这个YB42C0A83897-0006497CECEF4FA9-0-0 找不到了。看看下面信息是否有帮助分析问题。

obclient(root@sys)[oceanbase]> select * from dba_ob_servers;
+----------------+----------+------+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
| SVR_IP         | SVR_PORT | ID   | ZONE  | SQL_PORT | WITH_ROOTSERVER | STATUS | START_SERVICE_TIME         | STOP_TIME | BLOCK_MIGRATE_IN_TIME | CREATE_TIME                | MODIFY_TIME                | BUILD_VERSION                                                                             | LAST_OFFLINE_TIME |
+----------------+----------+------+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
| 192.168.56.151 |     2882 |    1 | zone1 |     2881 | YES             | ACTIVE | 2026-01-30 14:33:18.330231 | NULL      | NULL                  | 2026-01-26 11:19:43.849311 | 2026-01-30 14:33:18.637623 | 4.3.5.5_105010032025121323-524cb9015f8e905b5361bd410f99cf97c7df6557(Dec 13 2025 23:37:29) | NULL              |
| 192.168.56.152 |     2882 |    2 | zone2 |     2881 | NO              | ACTIVE | 2026-01-30 14:33:29.131781 | NULL      | NULL                  | 2026-01-26 11:19:43.917830 | 2026-01-30 14:33:30.865856 | 4.3.5.5_105010032025121323-524cb9015f8e905b5361bd410f99cf97c7df6557(Dec 13 2025 23:37:29) | NULL              |
| 192.168.56.153 |     2882 |    3 | zone3 |     2881 | NO              | ACTIVE | 2026-01-30 14:33:28.879708 | NULL      | NULL                  | 2026-01-26 11:19:44.000091 | 2026-01-30 14:33:30.922154 | 4.3.5.5_105010032025121323-524cb9015f8e905b5361bd410f99cf97c7df6557(Dec 13 2025 23:37:29) | NULL              |
+----------------+----------+------+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
3 rows in set (0.055 sec)

admin@obs03 log]$ 
[admin@obs03 log]$ grep 000649916BF10A06  rootservice.log
[2026-01-30 10:08:03.607548] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [19597][T1002_L0_G0][T1002][YB42C0A83897-000649916BF10A06-0-0] [lt=5] ObLSRecoveryStatHandler init success(this=0x7fbc52881090)
[admin@obs03 log]$

[admin@obs03 log]$ grep YB42C0A83897  rootservice.log
......

[2026-01-29 21:26:23.024037] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [100059][T1002_L0_G0][T1002][YB42C0A83897-0006497CECEF589A-0-0] [lt=4] ObLSRecoveryStatHandler init success(this=0x7fe570feb090)
[2026-01-29 21:26:35.541444] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [100148][T1001_HAService][T1001][YB42C0A83899-0006497CF9113529-0-0] [lt=0] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006497CECEF5884-0-0, tenant_id:1001, ls_id:{id:1}, result:-4184})
[2026-01-29 21:26:41.122115] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [100037][T1002_HAService][T1002][YB42C0A83899-0006497CF93177B4-0-0] [lt=3] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006497CECEF589A-0-0, tenant_id:1002, ls_id:{id:1}, result:-4184})
[2026-01-29 21:27:14.728730] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [100169][T1001_L0_G0][T1001][YB42C0A83897-0006497CECEF58A8-0-0] [lt=5] ObLSRecoveryStatHandler init success(this=0x7fe543635090)
[2026-01-29 21:27:24.997271] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [100059][T1002_L0_G0][T1002][YB42C0A83897-0006497CECEF58AC-0-0] [lt=4] ObLSRecoveryStatHandler init success(this=0x7fe570feb090)
[2026-01-30 08:19:27.868192] WDIAG [RS] update_tenant_info_cache (ob_tenant_info_loader.cpp:819) [10306][T1001_L0_G0][T1002][YB42C0A83897-0006498FF32598C4-0-0] [lt=39][errcode=-4023] my tenant_info is invalid, don't refresh(ret=-4023, ret="OB_EAGAIN", tenant_info={tenant_id:0, tenant_role:"INVALID", switchover_status:"INVALID", switchover_epoch:-1, sync_scn:0, replayable_scn:0, readable_scn:0, recovery_until_scn:0, log_mode:"INVALID", max_ls_id:-1, restore_data_mode:{restore data mode:"NORMAL"}}, ora_rowscn=0)
[2026-01-30 08:19:27.868216] WDIAG [RS] update_tenant_info_cache (ob_tenant_info_loader.cpp:720) [10306][T1001_L0_G0][T1002][YB42C0A83897-0006498FF32598C4-0-0] [lt=24][errcode=-4023] failed to update_tenant_info_cache(ret=-4023, ret="OB_EAGAIN", new_ora_rowscn=1769732367735177000, new_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1769693263146977007, replayable_scn:1769693263146977007, readable_scn:1769693263146977007, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}})
[2026-01-30 08:19:29.910984] WDIAG [RS] update_tenant_info_cache (ob_tenant_info_loader.cpp:819) [10304][T1001_L0_G0][T1002][YB42C0A83897-0006498FF32598CD-0-0] [lt=35][errcode=-4023] my tenant_info is invalid, don't refresh(ret=-4023, ret="OB_EAGAIN", tenant_info={tenant_id:0, tenant_role:"INVALID", switchover_status:"INVALID", switchover_epoch:-1, sync_scn:0, replayable_scn:0, readable_scn:0, recovery_until_scn:0, log_mode:"INVALID", max_ls_id:-1, restore_data_mode:{restore data mode:"NORMAL"}}, ora_rowscn=0)
[2026-01-30 08:19:29.911005] WDIAG [RS] update_tenant_info_cache (ob_tenant_info_loader.cpp:720) [10304][T1001_L0_G0][T1002][YB42C0A83897-0006498FF32598CD-0-0] [lt=21][errcode=-4023] failed to update_tenant_info_cache(ret=-4023, ret="OB_EAGAIN", new_ora_rowscn=1769732368944938000, new_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1769732367372931004, replayable_scn:1769732367372931004, readable_scn:1769732367372931003, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}})
[2026-01-30 08:19:30.917386] WDIAG [RS] update_tenant_info_cache (ob_tenant_info_loader.cpp:819) [10304][T1001_L0_G0][T1002][YB42C0A83897-0006498FF32598D3-0-0] [lt=5][errcode=-4023] my tenant_info is invalid, don't refresh(ret=-4023, ret="OB_EAGAIN", tenant_info={tenant_id:0, tenant_role:"INVALID", switchover_status:"INVALID", switchover_epoch:-1, sync_scn:0, replayable_scn:0, readable_scn:0, recovery_until_scn:0, log_mode:"INVALID", max_ls_id:-1, restore_data_mode:{restore data mode:"NORMAL"}}, ora_rowscn=0)
[2026-01-30 08:19:30.917409] WDIAG [RS] update_tenant_info_cache (ob_tenant_info_loader.cpp:720) [10304][T1001_L0_G0][T1002][YB42C0A83897-0006498FF32598D3-0-0] [lt=22][errcode=-4023] failed to update_tenant_info_cache(ret=-4023, ret="OB_EAGAIN", new_ora_rowscn=1769732370109974000, new_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1769732368882874000, replayable_scn:1769732368882874000, readable_scn:1769732368882874000, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}})
[2026-01-30 08:20:06.309117] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10304][T1001_L0_G0][T1001][YB42C0A83897-0006498FF4B598C1-0-0] [lt=5] ObLSRecoveryStatHandler init success(this=0x7f0f46cb1090)
[2026-01-30 08:20:06.968226] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10425][T1002_L0_G0][T1002][YB42C0A83897-0006498FF4B598C2-0-0] [lt=5] ObLSRecoveryStatHandler init success(this=0x7f0f433f5090)
[2026-01-30 08:21:06.654921] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [10283][T1001_HAService][T1001][YB42C0A83899-0006498FED446922-0-0] [lt=25] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006498FF4B598C1-0-0, tenant_id:1001, ls_id:{id:1}, result:-4184})
[2026-01-30 08:21:08.307030] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [10398][T1002_HAService][T1002][YB42C0A83899-0006498FEE946A24-0-0] [lt=4] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006498FF4B598C2-0-0, tenant_id:1002, ls_id:{id:1}, result:-4184})
[2026-01-30 08:21:39.806878] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10295][T1001_L0_G0][T1001][YB42C0A83897-0006498FF4B598DC-0-0] [lt=5] ObLSRecoveryStatHandler init success(this=0x7f0f46661090)
[2026-01-30 08:21:50.175186] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10413][T1002_L0_G0][T1002][YB42C0A83897-0006498FF4B598E0-0-0] [lt=7] ObLSRecoveryStatHandler init success(this=0x7f0f4b55f090)
[2026-01-30 08:22:06.995454] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [10283][T1001_HAService][T1001][YB42C0A83899-0006498FED446958-0-0] [lt=4] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006498FF4B598DC-0-0, tenant_id:1001, ls_id:{id:1}, result:-4184})
[2026-01-30 08:22:41.286082] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10304][T1001_L0_G0][T1001][YB42C0A83897-0006498FF4B598EE-0-0] [lt=4] ObLSRecoveryStatHandler init success(this=0x7f0f46661090)
[2026-01-30 08:23:07.539754] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [10283][T1001_HAService][T1001][YB42C0A83899-0006498FED44698D-0-0] [lt=3] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006498FF4B598EE-0-0, tenant_id:1001, ls_id:{id:1}, result:-4184})
[2026-01-30 08:23:08.466643] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [10398][T1002_HAService][T1002][YB42C0A83899-0006498FEE946B8E-0-0] [lt=3] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006498FF4B598E0-0-0, tenant_id:1002, ls_id:{id:1}, result:-4184})
[2026-01-30 08:23:42.730528] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10304][T1001_L0_G0][T1001][YB42C0A83897-0006498FF4B59900-0-0] [lt=6] ObLSRecoveryStatHandler init success(this=0x7f0f46661090)
[2026-01-30 08:23:53.189571] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10425][T1002_L0_G0][T1002][YB42C0A83897-0006498FF4B59904-0-0] [lt=4] ObLSRecoveryStatHandler init success(this=0x7f0f433f5090)
[2026-01-30 08:24:07.961137] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [10283][T1001_HAService][T1001][YB42C0A83899-0006498FED4469C1-0-0] [lt=5] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006498FF4B59900-0-0, tenant_id:1001, ls_id:{id:1}, result:-4184})
[2026-01-30 08:24:08.560414] INFO  [RS] report_to_meta_tenant (ob_disaster_recovery_task_utils.cpp:199) [10398][T1002_HAService][T1002][YB42C0A83899-0006498FEE946C46-0-0] [lt=6] send rpc success(leader="192.168.56.151:2882", arg={task_id:YB42C0A83897-0006498FF4B59904-0-0, tenant_id:1002, ls_id:{id:1}, result:-4184})
[2026-01-30 08:24:44.169668] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10304][T1001_L0_G0][T1001][YB42C0A83897-0006498FF4B59912-0-0] [lt=4] ObLSRecoveryStatHandler init success(this=0x7f0f46661090)
[2026-01-30 08:24:54.611562] INFO  [RS] init (ob_ls_recovery_stat_handler.cpp:170) [10425][T1002_L0_G0][T1002][YB42C0A83897-0006498FF4B59916-0-0] [lt=4] ObLSRecoveryStatHandler 
obclient(root@sys)[oceanbase]>
1 个赞
#10 
[admin@obs03 log]$ grep YB42C0A83897  observer*.log
。。。。。。
[2026-01-30 14:35:36.511206] INFO  [PALF] receive_config_log (palf_handle_impl.cpp:4169) [38453][T1001_L0_G1][T1001][YB42C0A83897-000649952B74022E-0-0] [lt=47] pre_check_for_mode_meta don't match, try fetch mode meta(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"192.168.56.153:2882", has_set_deleted:false}, server="192.168.56.151:2882", msg_proposal_id=5, prev_mode_pid=5, meta={version:3, proposal_id:5, prev:{version:1, config:{config_version:{proposal_id:5, config_seq:213}, log_sync_memberlist:2{server:"192.168.56.151:2882", timestamp:1, flag:0}{server:"192.168.56.152:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:5, config_seq:214}, log_sync_memberlist:2{server:"192.168.56.151:2882", timestamp:1, flag:0}{server:"192.168.56.152:2882", timestamp:1, flag:0}, log_sync_replica_num:2, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"192.168.56.153:2882", timestamp:1769754932391811, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:5, prev_lsn:{lsn:707666365}, prev_mode_pid:5})
[2026-01-30 14:35:36.519018] INFO  [COMMON] do_work (ob_tenant_dag_scheduler.cpp:348) [38194][T1002_TABLET_MI][T1002][YB42C0A83899-000649952C1F6AD1-0-0] [lt=7] task finish process(ret=0, start_time=1769754936500141, end_time=1769754936519016, runtime=18875, *this={"ObTabletFinishMigrationTask":"ObTabletFinishMigrationTask", this:0x7fa18bbc5170, ha_dag_net_ctx_:{ObIHADagNetCtx:{"ObIHADagNetCtx":"ObIHADagNetCtx", result_mgr:{result:0, retry_count:1, allow_retry:true, failed_task_id_list:[]}}, arg:{ls_id:{id:1}, type:"ADD_LS_OP", cluster_id:1769339529, priority:0, src:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, dst:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:2, replica_type:0, region:"default_region", memstore_percent:100}, data_src:{server:"0.0.0.0:0", timestamp:0, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, paxos_replica_number:3, tablet_id_array:[]}, local_clog_checkpoint_scn:{val:1769738900805738001, v:0}, src_ls_rebuild_seq:0, start_ts:1769754923190121, finish_ts:0, task_id:YB42C0A83897-000649952CC40217-0-0, minor_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, major_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, src_ls_meta_package:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754921040468004, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754936547154000, latest_log_ts:{val:1769754919903869001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, palf_meta:{prev_log_info:{log_id:387319, lsn:{lsn:134205286}, scn:{val:1769675031020399000, v:0}, log_proposal_id:2, accum_checksum:1409224050}, curr_lsn:{lsn:134209536}}, dup_ls_meta:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}}, check_tablet_info_cost_time:8806486}, copy_tablet_ctx_:{tablet_id:{id:50143}, status:0}, ls_:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:4}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:1, migration_status:1, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754933589226003, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754946547154000, latest_log_ts:{val:1769754929893311001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:5, palf_env_:0x7fa1bfff6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:3, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}})
[2026-01-30 14:35:36.519126] INFO  [COMMON] finish_dag_ (ob_tenant_dag_scheduler.cpp:2632) [38194][T1002_TABLET_MI][T1002][YB42C0A83899-000649952C1F6AD1-0-0] [lt=88] dag finished(dag_ret=0, runtime=116379, dag_cnt=16, dag_type_cnt=15, dag=0x7fa19134ed30, dag={ObIMigrationDag:{ObStorageHADag:{ObIDag:, ha_dag_net_ctx:{ObIHADagNetCtx:{"ObIHADagNetCtx":"ObIHADagNetCtx", result_mgr:{result:0, retry_count:1, allow_retry:true, failed_task_id_list:[]}}, arg:{ls_id:{id:1}, type:"ADD_LS_OP", cluster_id:1769339529, priority:0, src:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, dst:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:2, replica_type:0, region:"default_region", memstore_percent:100}, data_src:{server:"0.0.0.0:0", timestamp:0, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, paxos_replica_number:3, tablet_id_array:[]}, local_clog_checkpoint_scn:{val:1769738900805738001, v:0}, src_ls_rebuild_seq:0, start_ts:1769754923190121, finish_ts:0, task_id:YB42C0A8389-000649952CC40217-0-0, minor_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, major_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, src_ls_meta_package:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754921040468004, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754936547154000, latest_log_ts:{val:1769754919903869001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, palf_meta:{prev_log_info:{log_id:387319, lsn:{lsn:134205286}, scn:{val:1769675031020399000, v:0}, log_proposal_id:2, accum_checksum:1409224050}, curr_lsn:{lsn:134209536}}, dup_ls_meta:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}}, check_tablet_info_cost_time:8806486}, result_mgr:{result:0, retry_count:0, allow_retry:true, failed_task_id_list:[]}}, this:0x7fa19134ed30}, this:0x7fa19134ed30, copy_tablet_ctx_:{tablet_id:{id:50143}, status:0}, tablet_type_:1})
[2026-01-30 14:35:36.520587] INFO  [LIB] stat (utility.h:1238) [38627][T1002_L0_G1][T1002][YB42C0A83897-000649952D141370-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x7fa1b4feb6a0, extra_info="palf_id:1001")
[2026-01-30 14:35:36.520601] INFO  [LIB] stat (utility.h:1238) [38627][T1002_L0_G1][T1002][YB42C0A83897-000649952D141370-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=64940, this=0x7fa1b4feb8d0, extra_info="palf_id:1001")
[2026-01-30 14:35:36.520606] INFO  [PALF] sliding_cb (log_sliding_window.cpp:2278) [38627][T1002_L0_G1][T1002][YB42C0A83897-000649952D141370-0-0] [lt=4] [PALF STAT LOG TASK TIME](palf_id=1001, self="192.168.56.153:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=9, avg_log_submit_to_flush_time=6227, avg_log_submit_to_slide_time=64931)
[2026-01-30 14:35:36.520615] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2289) [38627][T1002_L0_G1][T1002][YB42C0A83897-000649952D141370-0-0] [lt=8][errcode=-4389] log_task life cost too much time(palf_id=1001, self="192.168.56.153:2882", log_id=565355, log_task={header:{begin_lsn:{lsn:68820221}, end_lsn:{lsn:68820343}, log_id:565355, min_scn:{val:1769754936097792003, v:0}, max_scn:{val:1769754936097792003, v:0}, data_len:66, proposal_id:5, prev_lsn:{lsn:68820099}, prev_proposal_id:5, committed_end_lsn:{lsn:68820221}, data_checksum:2222604892, accum_checksum:3953977875, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1769754936415016, freeze_ts:1769754936415016, submit_ts:1769754936415023, flushed_ts:1769754936418717, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:3694}, fs_cb_begin_ts=1769754936520574, log_life_time=105558)
[2026-01-30 14:35:36.523842] INFO  [COMMON] do_work (ob_tenant_dag_scheduler.cpp:348) [38195][T1002_TABLET_MI][T1002][YB42C0A83899-000649952C1F6ACE-0-0] [lt=6] task finish process(ret=0, start_time=1769754936489144, end_time=1769754936523841, runtime=34697, *this={"ObTabletFinishMigrationTask":"ObTabletFinishMigrationTask", this:0x7fa189558f70, ha_dag_net_ctx_:{ObIHADagNetCtx:{"ObIHADagNetCtx":"ObIHADagNetCtx", result_mgr:{result:0, retry_count:1, allow_retry:true, failed_task_id_list:[]}}, arg:{ls_id:{id:1}, type:"ADD_LS_OP", cluster_id:1769339529, priority:0, src:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, dst:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:2, replica_type:0, region:"default_region", memstore_percent:100}, data_src:{server:"0.0.0.0:0", timestamp:0, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, paxos_replica_number:3, tablet_id_array:[]}, local_clog_checkpoint_scn:{val:1769738900805738001, v:0}, src_ls_rebuild_seq:0, start_ts:1769754923190121, finish_ts:0, task_id:YB42C0A83897-000649952CC40217-0-0, minor_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, major_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, src_ls_meta_package:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754921040468004, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754936547154000, latest_log_ts:{val:1769754919903869001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, palf_meta:{prev_log_info:{log_id:387319, lsn:{lsn:134205286}, scn:{val:1769675031020399000, v:0}, log_proposal_id:2, accum_checksum:1409224050}, curr_lsn:{lsn:134209536}}, dup_ls_meta:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}}, check_tablet_info_cost_time:8806486}, copy_tablet_ctx_:{tablet_id:{id:60428}, status:0}, ls_:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:4}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:1, migration_status:1, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754933589226003, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754946547154000, latest_log_ts:{val:1769754929893311001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:5, palf_env_:0x7fa1bfff6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:3, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}})
[2026-01-30 14:35:36.523842] INFO  [COMMON] do_work (ob_tenant_dag_scheduler.cpp:348) [38197][T1002_TABLET_MI][T1002][YB42C0A83899-000649952C1F6ACD-0-0] [lt=7] task finish process(ret=0, start_time=1769754936497534, end_time=1769754936523841, runtime=26307, *this={"ObTabletFinishMigrationTask":"ObTabletFinishMigrationTask", this:0x7fa1877f2f70, ha_dag_net_ctx_:{ObIHADagNetCtx:{"ObIHADagNetCtx":"ObIHADagNetCtx", result_mgr:{result:0, retry_count:1, allow_retry:true, failed_task_id_list:[]}}, arg:{ls_id:{id:1}, type:"ADD_LS_OP", cluster_id:1769339529, priority:0, src:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, dst:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:2, replica_type:0, region:"default_region", memstore_percent:100}, data_src:{server:"0.0.0.0:0", timestamp:0, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, paxos_replica_number:3, tablet_id_array:[]}, local_clog_checkpoint_scn:{val:1769738900805738001, v:0}, src_ls_rebuild_seq:0, start_ts:1769754923190121, finish_ts:0, task_id:YB42C0A83897-000649952CC40217-0-0, minor_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, major_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, src_ls_meta_package:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754921040468004, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754936547154000, latest_log_ts:{val:1769754919903869001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, palf_meta:{prev_log_info:{log_id:387319, lsn:{lsn:134205286}, scn:{val:1769675031020399000, v:0}, log_proposal_id:2, accum_checksum:1409224050}, curr_lsn:{lsn:134209536}}, dup_ls_meta:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}}, check_tablet_info_cost_time:8806486}, copy_tablet_ctx_:{tablet_id:{id:102}, status:0}, ls_:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:4}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:1, migration_status:1, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754933589226003, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754946547154000, latest_log_ts:{val:1769754929893311001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:5, palf_env_:0x7fa1bfff6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:3, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}})
[2026-01-30 14:35:36.523944] INFO  [COMMON] finish_dag_ (ob_tenant_dag_scheduler.cpp:2632) [38195][T1002_TABLET_MI][T1002][YB42C0A83899-000649952C1F6ACE-0-0] [lt=80] dag finished(dag_ret=0, runtime=173528, dag_cnt=16, dag_type_cnt=15, dag=0x7fa19134cc30, dag={ObIMigrationDag:{ObStorageHADag:{ObIDag:, ha_dag_net_ctx:{ObIHADagNetCtx:{"ObIHADagNetCtx":"ObIHADagNetCtx", result_mgr:{result:0, retry_count:1, allow_retry:true, failed_task_id_list:[]}}, arg:{ls_id:{id:1}, type:"ADD_LS_OP", cluster_id:1769339529, priority:0, src:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, dst:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:2, replica_type:0, region:"default_region", memstore_percent:100}, data_src:{server:"0.0.0.0:0", timestamp:0, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, paxos_replica_number:3, tablet_id_array:[]}, local_clog_checkpoint_scn:{val:1769738900805738001, v:0}, src_ls_rebuild_seq:0, start_ts:1769754923190121, finish_ts:0, task_id:YB42C0A8389-000649952CC40217-0-0, minor_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, major_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, src_ls_meta_package:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754921040468004, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754936547154000, latest_log_ts:{val:1769754919903869001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, palf_meta:{prev_log_info:{log_id:387319, lsn:{lsn:134205286}, scn:{val:1769675031020399000, v:0}, log_proposal_id:2, accum_checksum:1409224050}, curr_lsn:{lsn:134209536}}, dup_ls_meta:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}}, check_tablet_info_cost_time:8806486}, result_mgr:{result:0, retry_count:0, allow_retry:true, failed_task_id_list:[]}}, this:0x7fa19134cc30}, this:0x7fa19134cc30, copy_tablet_ctx_:{tablet_id:{id:60428}, status:0}, tablet_type_:1})
[2026-01-30 14:35:36.523975] INFO  [SERVER] del_task (ob_sys_task_stat.cpp:193) [38195][T1002_TABLET_MI][T1002][YB42C0A83899-000649952C1F6ACE-0-0] [lt=29] succeed to del sys task(removed_task={start_time:1769754936350290, task_id:YB42C0A83899-000649952C1F6ACE-0-0, task_type:1, svr_ip:"192.168.56.153:2882", tenant_id:1002, is_cancel:false, comment:"TABLET_MIGRATION;tenant_id=1002;ls_id=1;tablet_id=60428;op_type=0;dag_net_task_id="YB42C0A83897-000649952CC40217-0-0",src=""192.168.56.153:2882"",dest=""192.168.56.153:2882"""})
[2026-01-30 14:35:36.524069] INFO  [SERVER] add_task (ob_sys_task_stat.cpp:139) [37792][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1] task id is valid, no need set new(ret=0, task={start_time:1769754936524061, task_id:YB42C0A83899-000649952C1F6AD7-0-0, task_type:1, svr_ip:"192.168.56.153:2882", tenant_id:1002, is_cancel:false, comment:"TABLET_MIGRATION;tenant_id=1002;ls_id=1;tablet_id=50133;op_type=0;dag_net_task_id="YB42C0A83897-000649952CC40217-0-0",src=""192.168.56.153:2882"",dest=""192.168.56.153:2882"""})
[2026-01-30 14:35:36.524075] INFO  [SERVER] add_task (ob_sys_task_stat.cpp:164) [37792][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1769754936524061, task_id:YB42C0A83899-000649952C1F6AD7-0-0, task_type:1, svr_ip:"192.168.56.153:2882", tenant_id:1002, is_cancel:false, comment:"TABLET_MIGRATION;tenant_id=1002;ls_id=1;tablet_id=50133;op_type=0;dag_net_task_id="YB42C0A83897-000649952CC40217-0-0",src=""192.168.56.153:2882"",dest=""192.168.56.153:2882"""})
[2026-01-30 14:35:36.525441] INFO  [COMMON] finish_dag_ (ob_tenant_dag_scheduler.cpp:2632) [38197][T1002_TABLET_MI][T1002][YB42C0A83899-000649952C1F6ACD-0-0] [lt=97] dag finished(dag_ret=0, runtime=203081, dag_cnt=16, dag_type_cnt=15, dag=0x7fa19134ba30, dag={ObIMigrationDag:{ObStorageHADag:{ObIDag:, ha_dag_net_ctx:{ObIHADagNetCtx:{"ObIHADagNetCtx":"ObIHADagNetCtx", result_mgr:{result:0, retry_count:1, allow_retry:true, failed_task_id_list:[]}}, arg:{ls_id:{id:1}, type:"ADD_LS_OP", cluster_id:1769339529, priority:0, src:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, dst:{server:"192.168.56.153:2882", timestamp:1769754922317794, flag:2, replica_type:0, region:"default_region", memstore_percent:100}, data_src:{server:"0.0.0.0:0", timestamp:0, flag:0, replica_type:0, region:"default_region", memstore_percent:100}, paxos_replica_number:3, tablet_id_array:[]}, local_clog_checkpoint_scn:{val:1769738900805738001, v:0}, src_ls_rebuild_seq:0, start_ts:1769754923190121, finish_ts:0, task_id:YB42C0A8389-000649952CC40217-0-0, minor_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, major_src:{src_addr:"192.168.56.152:2882", cluster_id:1769339529}, src_ls_meta_package:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1769738900805738001, v:0}, clog_base_lsn:{lsn:134209536}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1769754921040468004, v:0}, tablet_change_checkpoint_scn:{val:1769738909018115003, v:0}, all_id_meta:{id_meta:[{limited_id:1769754936547154000, latest_log_ts:{val:1769754919903869001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1769754786547154002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, palf_meta:{prev_log_info:{log_id:387319, lsn:{lsn:134205286}, scn:{val:1769675031020399000, v:0}, log_proposal_id:2, accum_checksum:1409224050}, curr_lsn:{lsn:134209536}}, dup_ls_meta:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}}, check_tablet_info_cost_time:8806486}, result_mgr:{result:0, retry_count:0, allow_retry:true, failed_task_id_list:[]}}, this:0x7fa19134ba30}, this:0x7fa19134ba30, copy_tablet_ctx_:{tablet_id:{id:102}, status:0}, tablet_type_:1})
[2026-01-30 14:35:36.525522] INFO  [SERVER] del_task (ob_sys_task_stat.cpp:193) [38197][T1002_TABLET_MI][T1002][YB42C0A83899-000649952C1F6ACD-0-0] [lt=78] succeed to del sys task(removed_task={start_time:1769754936322219, task_id:YB42C0A83899-000649952C1F6ACD-0-0, task_type:1, svr_ip:"192.168.56.153:2882", tenant_id:1002, is_cancel:false, comment:"TABLET_MIGRATION;tenant_id=1002;ls_id=1;tablet_id=102;op_type=0;dag_net_task_id="YB42C0A83897-000649952CC40217-0-0",src=""192.168.56.153:2882"",dest=""192.168.56.153:2882"""})
[2026-01-30 14:35:36.525596] INFO  [SERVER] add_task (ob_sys_task_stat.cpp:139) [37792][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1] task id is valid, no need set new(ret=0, task={start_time:1769754936525589, task_id:YB42C0A83899-000649952C1F6AD8-0-0, task_type:1, svr_ip:"192.168.56.153:2882", tenant_id:1002, is_cancel:false, comment:"TABLET_MIGRATION;tenant_id=1002;ls_id=1;tablet_id=60523;op_type=0;dag_net_task_id="YB42C0A83897-000649952CC40217-0-
1 个赞
#11 
有命令停止这个ALTER_TENANT_LOCALITY job吗？
obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
| JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                         | EXTRA_INFO                                                                              | RS_SVR_IP      | RS_SVR_PORT |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
|      4 | ALTER_TENANT_LOCALITY | SUCCESS    |           0 |      100 | 2026-01-29 09:55:07.663139 | 2026-01-29 09:55:38.787132 |      1002 | ALTER TENANT `tnt_20260126` LOCALITY = 'FULL{1}@zone1, FULL{1}@zone2'            | FROM: 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3', TO: 'FULL{1}@zone1, FULL{1}@zone2' | 192.168.56.151 |        2882 |
|      5 | ALTER_TENANT_LOCALITY | INPROGRESS |        NULL |        0 | 2026-01-29 19:16:34.121401 | 2026-01-29 19:16:34.121401 |      1002 | ALTER TENANT tnt_20260126 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3" | FROM: 'FULL{1}@zone1, FULL{1}@zone2', TO: 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3' | 192.168.56.151 |        2882 |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
2 rows in set (0.010 sec)
1 个赞
#12 
根据-4184错误码，修改datafile参数后解决。

![image|690x81](upload://1UcKP11FfnvYRLXRlW70kIVgk96.png)

obclient(root@sys)[oceanbase]>  SHOW PARAMETERS like  'datafile%';    
+-------+----------+----------------+----------+--------------------------+-----------+-------+-------------------------------------------------------------------------------+---------+---------+---------+-------------------+---------------+-----------+
| zone  | svr_type | svr_ip         | svr_port | name                     | data_type | value | info                                                                          | section | scope   | source  | edit_level        | default_value | isdefault |
+-------+----------+----------------+----------+--------------------------+-----------+-------+-------------------------------------------------------------------------------+---------+---------+---------+-------------------+---------------+-----------+
| zone1 | observer | 192.168.56.151 |     2882 | datafile_disk_percentage | INT       | 0     | the percentage of disk space used by the data files. Range: [0,99] in integer | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         1 |
| zone1 | observer | 192.168.56.151 |     2882 | datafile_maxsize         | CAPACITY  | 2G    | the auto extend max size. Range: [0, +∞)                                      | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         0 |
| zone1 | observer | 192.168.56.151 |     2882 | datafile_next            | CAPACITY  | 0     | the auto extend step. Range: [0, +∞)                                          | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         1 |
| zone1 | observer | 192.168.56.151 |     2882 | datafile_size            | CAPACITY  | 1G    | size of the data file. Range: [0, +∞)                                         | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0M            |         0 |
| zone3 | observer | 192.168.56.153 |     2882 | datafile_disk_percentage | INT       | 0     | the percentage of disk space used by the data files. Range: [0,99] in integer | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         1 |
| zone3 | observer | 192.168.56.153 |     2882 | datafile_maxsize         | CAPACITY  | 2G    | the auto extend max size. Range: [0, +∞)                                      | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         0 |
| zone3 | observer | 192.168.56.153 |     2882 | datafile_next            | CAPACITY  | 0     | the auto extend step. Range: [0, +∞)                                          | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         1 |
| zone3 | observer | 192.168.56.153 |     2882 | datafile_size            | CAPACITY  | 1G    | size of the data file. Range: [0, +∞)                                         | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0M            |         0 |
| zone2 | observer | 192.168.56.152 |     2882 | datafile_disk_percentage | INT       | 0     | the percentage of disk space used by the data files. Range: [0,99] in integer | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         1 |
| zone2 | observer | 192.168.56.152 |     2882 | datafile_maxsize         | CAPACITY  | 2G    | the auto extend max size. Range: [0, +∞)                                      | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         0 |
| zone2 | observer | 192.168.56.152 |     2882 | datafile_next            | CAPACITY  | 0     | the auto extend step. Range: [0, +∞)                                          | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0             |         1 |
| zone2 | observer | 192.168.56.152 |     2882 | datafile_size            | CAPACITY  | 1G    | size of the data file. Range: [0, +∞)                                         | SSTABLE | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | 0M            |         0 |
+-------+----------+----------------+----------+--------------------------+-----------+-------+-------------------------------------------------------------------------------+---------+---------+---------+-------------------+---------------+-----------+
12 rows in set (0.015 sec)

obclient(root@sys)[oceanbase]> 


obclient(root@sys)[oceanbase]> SELECT SVR_IP, CONCAT(ROUND((DATA_DISK_CAPACITY)/1024/1024/1024,1),' GB') AS DATA_DISK_CAPACITY,
    -> CONCAT(ROUND((DATA_DISK_ALLOCATED)/1024/1024/1024,1),' GB') AS DATA_DISK_ALLOCATED,
    -> CONCAT(ROUND((DATA_DISK_IN_USE)/1024/1024/1024,1),' GB') AS DATA_DISK_IN_USE FROM GV$OB_SERVERS;
+----------------+--------------------+---------------------+------------------+
| SVR_IP         | DATA_DISK_CAPACITY | DATA_DISK_ALLOCATED | DATA_DISK_IN_USE |
+----------------+--------------------+---------------------+------------------+
| 192.168.56.153 | 2.0 GB             | 1.0 GB              | 0.2 GB           |
| 192.168.56.152 | 2.0 GB             | 1.0 GB              | 0.4 GB           |
| 192.168.56.151 | 2.0 GB             | 1.0 GB              | 0.5 GB           |
+----------------+--------------------+---------------------+------------------+
3 rows in set (0.183 sec)


ALTER SYSTEM SET datafile_maxsize = '5G';
ALTER SYSTEM SET datafile_next = '500M';

obclient(root@sys)[oceanbase]> 

obclient(root@sys)[oceanbase]> SELECT SVR_IP, CONCAT(ROUND((DATA_DISK_CAPACITY)/1024/1024/1024,1),' GB') AS DATA_DISK_CAPACITY,
    -> CONCAT(ROUND((DATA_DISK_ALLOCATED)/1024/1024/1024,1),' GB') AS DATA_DISK_ALLOCATED,
    -> CONCAT(ROUND((DATA_DISK_IN_USE)/1024/1024/1024,1),' GB') AS DATA_DISK_IN_USE FROM GV$OB_SERVERS;
+----------------+--------------------+---------------------+------------------+
| SVR_IP         | DATA_DISK_CAPACITY | DATA_DISK_ALLOCATED | DATA_DISK_IN_USE |
+----------------+--------------------+---------------------+------------------+
| 192.168.56.152 | 5.0 GB             | 1.5 GB              | 0.4 GB           |
| 192.168.56.153 | 5.0 GB             | 3.5 GB              | 0.4 GB           |
| 192.168.56.151 | 5.0 GB             | 2.0 GB              | 0.5 GB           |
+----------------+--------------------+---------------------+------------------+
3 rows in set (0.006 sec)

obclient(root@sys)[oceanbase]> 
obclient(root@sys)[oceanbase]> SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';                                         +--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
| JOB_ID | JOB_TYPE              | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME                 | MODIFY_TIME                | TENANT_ID | SQL_TEXT                                                                         | EXTRA_INFO                                                                              | RS_SVR_IP      | RS_SVR_PORT |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
|      4 | ALTER_TENANT_LOCALITY | SUCCESS    |           0 |      100 | 2026-01-29 09:55:07.663139 | 2026-01-29 09:55:38.787132 |      1002 | ALTER TENANT `tnt_20260126` LOCALITY = 'FULL{1}@zone1, FULL{1}@zone2'            | FROM: 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3', TO: 'FULL{1}@zone1, FULL{1}@zone2' | 192.168.56.151 |        2882 |
|      5 | ALTER_TENANT_LOCALITY | SUCCESS    |           0 |      100 | 2026-01-29 19:16:34.121401 | 2026-01-30 16:50:20.514642 |      1002 | ALTER TENANT tnt_20260126 locality="FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3" | FROM: 'FULL{1}@zone1, FULL{1}@zone2', TO: 'FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3' | 192.168.56.151 |        2882 |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+----------------+-------------+
2 rows in set (0.008 sec)

obclient(root@sys)[oceanbase]>
1 个赞
#13

怀疑就是设置的数据空间不足导致的 你没有查看数据盘使用量是多少 没有办法判断 调整以后可以就行了

1 个赞