旁路导入回滚阻塞整库ddl

OceanBase 技术问题
#1

【 使用环境 】生产环境
【 OB or 其他组件 】OB
【 使用版本 】4.3.5
【问题描述】清晰明确描述问题
mysql> select * from oceanbase.GV$SESSION_LONGOPS;
±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
| SID | TRACE_ID | OPNAME | TARGET | SVR_IP | SVR_PORT | START_TIME | ELAPSED_SECONDS | TIME_REMAINING | LAST_UPDATE_TIME | MESSAGE |
±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
| -1 | Y100000ADF6316-000649E259E1B98E-0-0 | direct load | 578222 | xx.xx.xx.22 | 2882 | 2026-02-03 10:39:40 | 102074 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694363935, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649EDBB42C88B-0-0 | direct load | 579191 | xx.xx.xx.22 | 2882 | 2026-02-04 00:14:19 | 53196 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 734956301, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E260771A97-0-0 | direct load | 578223 | xx.xx.xx.22 | 2882 | 2026-02-03 10:41:30 | 101964 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694424141, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E26347D66B-0-0 | direct load | 578227 | xx.xx.xx.22 | 2882 | 2026-02-03 10:42:18 | 101917 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694450029, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E26215D678-0-0 | direct load | 578224 | xx.xx.xx.22 | 2882 | 2026-02-03 10:41:57 | 101937 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694439037, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24F6AD885-0-0 | direct load | 578205 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:45 | 102250 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694268041, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E2638DC8AE-0-0 | direct load | 578230 | xx.xx.xx.22 | 2882 | 2026-02-03 10:42:22 | 101912 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694452557, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24FDC9D8E-0-0 | direct load | 578214 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:53 | 102242 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694271182, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24F71068E-0-0 | direct load | 578206 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:45 | 102249 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694268145, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E2636D967D-0-0 | direct load | 578229 | xx.xx.xx.22 | 2882 | 2026-02-03 10:42:20 | 101914 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694451392, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E259BCD054-0-0 | direct load | 578220 | xx.xx.xx.22 | 2882 | 2026-02-03 10:39:37 | 102077 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694362512, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24FD7A41D-0-0 | direct load | 578213 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:52 | 102242 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694271047, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E262626BF1-0-0 | direct load | 578225 | xx.xx.xx.22 | 2882 | 2026-02-03 10:42:03 | 101932 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694441780, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24FACEE80-0-0 | direct load | 578208 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:49 | 102246 | 0 | 2026-02-04 15:00:54 | TENANT_ID: 1002, TASK_ID: 694269593, STATUS: CLEAN ON FAIL |
±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
14 rows in set (0.07 sec)

mysql> ALTER SYSTEM CANCEL TASK ‘Y100000ADF6316-000649E24F6AD885-0-0’;
Query OK, 0 rows affected (0.01 sec)

mysql> select * from oceanbase.GV$SESSION_LONGOPS where TRACE_ID=‘Y100000ADF6316-000649E24F6AD885-0-0’;
±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
| SID | TRACE_ID | OPNAME | TARGET | SVR_IP | SVR_PORT | START_TIME | ELAPSED_SECONDS | TIME_REMAINING | LAST_UPDATE_TIME | MESSAGE |
±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
| -1 | Y100000ADF6316-000649E24F6AD885-0-0 | direct load | 578205 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:45 | 102305 | 0 | 2026-02-04 15:01:50 | TENANT_ID: 1002, TASK_ID: 694268041, STATUS: CLEAN ON FAIL |
±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
1 row in set (0.07 sec)

mysql> SELECT * FROM oceanbase.__all_virtual_ddl_task_status where trace_id=‘Y100000ADF6316-000649E24F6AD885-0-0’\G;
*************************** 1. row ***************************
tenant_id: 1002
task_id: 694268041
gmt_create: 2026-02-03 10:36:44.770537
gmt_modified: 2026-02-04 11:23:59.805079
object_id: 508104
target_object_id: 578205
ddl_type: 1011
schema_version: 1770086204716328
parent_task_id: 0
trace_id: Y100000ADF6316-000649E24F6AD885-0-0
status: 99
snapshot_version: 0
task_version: 1
execution_id: -1
ddl_stmt_str:
ret_code: -4072

目前整个租户不管涉不涉及这个表的ddl，全部卡死，包括建库，建表等操作
为啥旁路导入回滚，会影响整个租户的ddl

【复现路径】问题出现前后相关操作
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接（右键跳转查看）：

【SOP系列 22 】——故障诊断第一步(自助诊断和诊断信息收集)

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！

7 个赞
#3

针对您提到的旁路导入回滚操作导致整个租户的 DDL 操作卡死的问题，可以利用 obdiag 工具来帮助我们收集日志并进行根因分析。下面是具体的步骤和建议：

日志采集场景: DDL 过程中报磁盘空间不足的问题（虽然您的问题是 DDL 卡死，但这里没有特定的 DDL 卡死场景，因此选择与 DDL 相关的场景作为采集依据）
日志采集命令: obdiag gather scene run --scene=observer.ddl_disk_full

根因分析场景: 卡合并场景（这里选择与卡住相关的场景进行根因分析）
根因分析命令: obdiag rca run --scene=major_hold

请您先执行上述命令，将收集到的日志压缩包发送给我们，以便进一步分析问题的原因。同时，为了更准确地定位问题，请您提供以下信息：

  1. 旁路导入的具体操作内容是什么？
  2. 何时开始出现 DDL 操作卡死的现象？
  3. 当前租户的 DDL 操作是否有较高的并发量？
  4. 系统资源（如 CPU、内存、磁盘 I/O）在 DDL 操作卡死期间的表现如何？
  5. 是否有其他异常日志或错误提示？

希望这些信息能够帮助我们更快地找到问题的根源。感谢您的配合！

附上敏捷诊断工具 obdiag 使用帮助链接

1 个赞
#4

你做的旁路导入的语句发一下 你是不是开并发了 开的很高呀 资源被吃的很厉害呀 目前恢复了么？

#5

±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
| SID | TRACE_ID | OPNAME | TARGET | SVR_IP | SVR_PORT | START_TIME | ELAPSED_SECONDS | TIME_REMAINING | LAST_UPDATE_TIME | MESSAGE |
±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
| -1 | Y100000ADF6316-000649E259E1B98E-0-0 | direct load | 578222 | xx.xx.xx.22 | 2882 | 2026-02-03 10:39:40 | 169789 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694363935, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649EDBB42C88B-0-0 | direct load | 579191 | xx.xx.xx.22 | 2882 | 2026-02-04 00:14:19 | 120910 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 734956301, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E260771A97-0-0 | direct load | 578223 | xx.xx.xx.22 | 2882 | 2026-02-03 10:41:30 | 169679 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694424141, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E26347D66B-0-0 | direct load | 578227 | xx.xx.xx.22 | 2882 | 2026-02-03 10:42:18 | 169632 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694450029, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E26215D678-0-0 | direct load | 578224 | xx.xx.xx.22 | 2882 | 2026-02-03 10:41:57 | 169652 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694439037, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24F6AD885-0-0 | direct load | 578205 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:45 | 169964 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694268041, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E2638DC8AE-0-0 | direct load | 578230 | xx.xx.xx.22 | 2882 | 2026-02-03 10:42:22 | 169627 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694452557, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24FDC9D8E-0-0 | direct load | 578214 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:53 | 169957 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694271182, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24F71068E-0-0 | direct load | 578206 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:45 | 169964 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694268145, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E2636D967D-0-0 | direct load | 578229 | xx.xx.xx.22 | 2882 | 2026-02-03 10:42:20 | 169629 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694451392, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E259BCD054-0-0 | direct load | 578220 | xx.xx.xx.22 | 2882 | 2026-02-03 10:39:37 | 169792 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694362512, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24FD7A41D-0-0 | direct load | 578213 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:52 | 169957 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694271047, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E262626BF1-0-0 | direct load | 578225 | xx.xx.xx.22 | 2882 | 2026-02-03 10:42:03 | 169647 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694441780, STATUS: CLEAN ON FAIL |
| -1 | Y100000ADF6316-000649E24FACEE80-0-0 | direct load | 578208 | xx.xx.xx.22 | 2882 | 2026-02-03 10:36:49 | 169960 | 0 | 2026-02-05 09:49:29 | TENANT_ID: 1002, TASK_ID: 694269593, STATUS: CLEAN ON FAIL |
±----±------------------------------------±------------±-------±--------------±---------±--------------------±----------------±---------------±--------------------±-----------------------------------------------------------+
14 rows in set (0.07 sec)

#6

现在在回滚，滚了2天都没滚完，目前cpu、io没什么压力

#7

了解学习下

#8

– 查看任务详细信息

SELECT task_id, object_id, target_object_id, ret_code, message

FROM oceanbase.__all_virtual_ddl_task_status

WHERE status = 99 AND tenant_id = 1002;

#9

observer.log.20260204094414434:[2026-02-04 09:40:16.780559] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:201) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=20][errcode=-6005] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-6005, remote_addr=“10.140.6.25:2882”)
observer.log.20260204094414434:[2026-02-04 09:40:16.780582] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:47) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=22][errcode=-6005] failed to setup first scanner(ret=-6005)
observer.log.20260204094414434:[2026-02-04 09:40:16.780600] WDIAG [SQL] open (ob_result_set.cpp:139) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=-6005] open result set failed(ret=-6005)
observer.log.20260204094414434:[2026-02-04 09:40:16.780617] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=15][errcode=-6005] open result set failed(ret=-6005)
observer.log.20260204094414434:[2026-02-04 09:40:16.780635] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=-6005] result set open failed(ret=-6005, executor={ObIExecutor:, sql:“SELECT status, execution_id, ret_code, snapshot_version FROM __all_ddl_task_status WHERE task_id = 734956301 FOR UPDATE”})
observer.log.20260204094414434:[2026-02-04 09:40:16.780704] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=1][errcode=-6005] execute failed(ret=-6005, tenant_id=1002, executor={ObIExecutor:, sql:“SELECT status, execution_id, ret_code, snapshot_version FROM __all_ddl_task_status WHERE task_id = 734956301 FOR UPDATE”}, retry_cnt=0, local_sys_schema_version=1769590007612728, local_tenant_schema_version=1770135258697984)
observer.log.20260204094414434:[2026-02-04 09:40:16.780998] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=1][errcode=-6005] result set close failed(ret=-6005)
observer.log.20260204094414434:[2026-02-04 09:40:16.781022] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=23][errcode=-6005] result set close failed(ret=-6005)
observer.log.20260204094414434:[2026-02-04 09:40:16.781039] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=-6005] failed to close result(close_ret=-6005, ret=-6005)
observer.log.20260204094414434:[2026-02-04 09:40:16.781062] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [1289702][T1_DDLTaskExecu][T1002][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=-6005] failed to process record(executor={ObIExecutor:, sql:“SELECT status, execution_id, ret_code, snapshot_version FROM __all_ddl_task_status WHERE task_id = 734956301 FOR UPDATE”}, record_ret=-6005, ret=-6005)

observer.log.20260204113230211:[2026-02-04 11:23:59.031984] INFO [SHARE] add_event (ob_event_history_table_operator.h:266) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=21] event table add task(ret=0, event_table_name=“all_rootservice_event_history", sql=INSERT INTO all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1770175439031949), ‘ddl scheduler’, ‘release snapshot finish’, ‘tenant_id’, 1002, ‘ret’, -4072, ‘trace_id’, ‘Y100000ADF6316-000649EDBB42C88B-0-0’, ‘task_id’, 734956301, ‘task_status’, ‘FAIL’, ‘schema_version’, 1770135258610064, ‘object_id:506840, target_object_id:579191’, ‘xx.xx.xx.22’, 2882))
observer.log.20260204113230211:[2026-02-04 11:23:59.032001] INFO [SHARE] release_snapshot (ob_ddl_common.cpp:1759) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=15] release snapshot finished(ret=0, snapshot_version=1770135259166421000, table_id=506840, target_table_id=579191, schema_version=1770135258610064, ddl_event_info={addr:“xx.xx.xx.22:2882”, event_ts:1770175439032000, sub_id:0, trace_id:Y100000ADF6316-000649EDBB42C88B-0-0, parent_trace_id_:Y100000ADF6316-000649EDBB42C88B-0-0})
observer.log.20260204113230211:[2026-02-04 11:23:59.043354] INFO [SHARE.SCHEMA] retrieve_table_schema (ob_schema_retrieve_utils.ipp:387) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=27] retrieve table schema succeed(*table_schema={simple_table_schema:{tenant_id:1002, database_id:505233, tablegroup_id:18446744073709551615, table_id:506846, association_table_id:18446744073709551615, in_offline_ddl_white_list:false, table_name:”__idx_506840_idx_allocate_flag", session_id:0, index_type:1, table_type:5, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, pk_exists:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0, mv_container_table_flag:0, mv_available_flag:0, table_referenced_by_mv_flag:0, mv_enable_query_rewrite_flag:0, mv_on_query_computation_flag:0, ddl_table_ignore_sync_cdc_flag:0, table_organization_mode:0}, tablespace_id:18446744073709551615, data_table_id:506840, name_casemode:-1, schema_version:1769168976291016, part_level:0, part_option:{part_func_type:0, part_func_expr:"", part_num:1, auto_part:false, auto_part_size:0}, sub_part_option:{part_func_type:0, part_func_expr:"", part_num:0, auto_part:false, auto_part_size:0}, partition_num:0, def_subpartition_num:0, partition_array:null, partition_array_capacity:0, def_subpartition_array:null, hidden_partition_array:null, index_status:2, duplicate_scope:0, encryption:"", encrypt_key:"", master_key_id:18446744073709551615, sub_part_template_flags:0, get_tablet_id():{id:1152921504606848600}, max_dependency_version:-1, object_status:1, is_force_view:false, truncate_version:-1, duplicate_read_consistency:0, storage_cache_policy_type:4, with_dynamic_partition_policy:false}, max_used_column_id:44, sess_active_time:0, rowkey_column_num:2, index_column_num:1, rowkey_split_pos:0, block_size:16384, is_use_bloomfilter:false, progressive_merge_num:0, tablet_size:134217728, pctfree:0, load_type:0, index_using_type:0, def_type:1, charset_type:2, collation_type:46, index_status:2, partition_status:0, code_version:1, comment:"", pk_comment:"", create_host:"", tablegroup_name:"", compressor_type:6, row_store_type:1, store_format:3, expire_info:"", view_schema:{view_definition:"", check_option:“none”, is_updatable:“false”, is_materialized:“false”, character_set_client:0, collation_connection:“invalid_type”}, autoinc_column_id:0, auto_increment:1, read_only:false, simple_index_infos:[], base_table_ids:[], rowkey_info:{columns:null, capacity:0}, partition_key_info:{columns:null, capacity:0}, column_cnt:0, table_dop:1, constraints:null, column_array:null, aux_vp_tid_array:[], index_info:{columns:null, capacity:0}, define_user_id:18446744073709551615, aux_lob_meta_tid:18446744073709551615, aux_lob_piece_tid:18446744073709551615, name_generated_type:1, lob_inrow_threshold:8192, is_column_store_supported:true, max_used_column_group_id:1000, column_group_cnt:0, column_group_array:null, mlog_tid:18446744073709551615, auto_increment_cache_size:0, local_session_vars:{}, index_params:"", exec_env:"", storage_cache_policy:"{“GLOBAL”:“AUTO”}", merge_engine_type:0, semistruct_encoding_type:{mode:0, reserved:0}, dynamic_partition_policy:""}, is_deleted=false)
observer.log.20260204113230211:[2026-02-04 11:23:59.069139] INFO [SHARE.SCHEMA] get_table_schema (ob_schema_service_sql_impl.cpp:6241) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=19] get not core table schema succeed(schema_status={tenant_id:1002, snapshot_timestamp:-1, readable_schema_version:-1}, table_id=506846, table_schema->get_table_name_str()=__idx_506840_idx_allocate_flag)

observer.log.20260204113230211:[2026-02-04 11:23:59.366741] INFO [SHARE.SCHEMA] get_table_schema (ob_schema_service_sql_impl.cpp:6241) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=1] get not core table schema succeed(schema_status={tenant_id:1002, snapshot_timestamp:-1, readable_schema_version:-1}, table_id=506933, table_schema->get_table_name_str()=__idx_506840_idx_wms_flag_allocate_flag_extra2_next_time)
observer.log.20260204113230211:[2026-02-04 11:24:00.723211] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=24][errcode=-4122] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4122, timeout=1000000000)
observer.log.20260204113230211:[2026-02-04 11:24:00.723241] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=28][errcode=-4122]
observer.log.20260204114116766:[2026-02-04 11:40:43.386702] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289702][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=8][errcode=-4122] REACH SYSLOG RATE LIMIT [bandwidth]

observer.log.20260204125447372:[2026-02-04 12:47:24.533071] WDIAG [RPC] send (ob_poc_rpc_proxy.h:171) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4012] sync rpc execute fail(ret=-4012, addr=“xx.xx.xx.22:2882”, pcode=5144, timeout=1000000000)
observer.log.20260204125447372:[2026-02-04 12:47:24.533130] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=15][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770180444533121, ref_timestamp=1770179444530593, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204125447372:[2026-02-04 12:47:24.533152] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289701][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=0] long time to hold one guard(schema_mgr=0x7ef7319a4060, tenant_id=1002, version=1770176442624264, cur_timestamp=1770180444533147, ref_timestamp=1770179444530593, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204131035347:[2026-02-04 13:04:05.352073] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4122] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4122, timeout=1000000000)
observer.log.20260204131035347:[2026-02-04 13:04:05.352112] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=36][errcode=-4122]
observer.log.20260204131035347:[2026-02-04 13:04:05.352157] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=13][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770181445352147, ref_timestamp=1770180445457812, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204131035347:[2026-02-04 13:04:05.352178] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=0] long time to hold one guard(schema_mgr=0x7ef7319e4950, tenant_id=1002, version=1770180445314536, cur_timestamp=1770181445352173, ref_timestamp=1770180445457812, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204134228248:[2026-02-04 13:37:25.019097] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=7][errcode=-4012] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4012, timeout=1000000000)
observer.log.20260204134228248:[2026-02-04 13:37:25.019125] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=25][errcode=-4012]
observer.log.20260204134228248:[2026-02-04 13:37:25.019161] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770183445019152, ref_timestamp=1770182445083701, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204134228248:[2026-02-04 13:37:25.019185] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=19][errcode=0] long time to hold one guard(schema_mgr=0x7ef7319e4950, tenant_id=1002, version=1770180445314536, cur_timestamp=1770183445019181, ref_timestamp=1770182445083701, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204135753145:[2026-02-04 13:54:05.106514] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4012] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4012, timeout=1000000000)
observer.log.20260204135753145:[2026-02-04 13:54:05.106553] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=36][errcode=-4012]
observer.log.20260204135753145:[2026-02-04 13:54:05.106613] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770184445106585, ref_timestamp=1770183445130529, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204135753145:[2026-02-04 13:54:05.106635] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=0] long time to hold one guard(schema_mgr=0x7ef7319e4950, tenant_id=1002, version=1770180445314536, cur_timestamp=1770184445106629, ref_timestamp=1770183445130529, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)

observer.log.20260204131035347:[2026-02-04 13:04:05.352073] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4122] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4122, timeout=1000000000)
observer.log.20260204131035347:[2026-02-04 13:04:05.352112] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=36][errcode=-4122]
observer.log.20260204131035347:[2026-02-04 13:04:05.352157] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=13][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770181445352147, ref_timestamp=1770180445457812, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204131035347:[2026-02-04 13:04:05.352178] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=0] long time to hold one guard(schema_mgr=0x7ef7319e4950, tenant_id=1002, version=1770180445314536, cur_timestamp=1770181445352173, ref_timestamp=1770180445457812, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204134228248:[2026-02-04 13:37:25.019097] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=7][errcode=-4012] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4012, timeout=1000000000)
observer.log.20260204134228248:[2026-02-04 13:37:25.019125] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=25][errcode=-4012]
observer.log.20260204134228248:[2026-02-04 13:37:25.019161] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770183445019152, ref_timestamp=1770182445083701, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204134228248:[2026-02-04 13:37:25.019185] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=19][errcode=0] long time to hold one guard(schema_mgr=0x7ef7319e4950, tenant_id=1002, version=1770180445314536, cur_timestamp=1770183445019181, ref_timestamp=1770182445083701, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204135753145:[2026-02-04 13:54:05.106514] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4012] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4012, timeout=1000000000)
observer.log.20260204135753145:[2026-02-04 13:54:05.106553] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=36][errcode=-4012]
observer.log.20260204135753145:[2026-02-04 13:54:05.106613] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770184445106585, ref_timestamp=1770183445130529, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204135753145:[2026-02-04 13:54:05.106635] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=0] long time to hold one guard(schema_mgr=0x7ef7319e4950, tenant_id=1002, version=1770180445314536, cur_timestamp=1770184445106629, ref_timestamp=1770183445130529, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
[apps@gd16c-db-115022-db-ob log]$ grep ‘Y100000ADF6316-000649EDBB42C88B-0-0’ observer.log.2026020414*|grep -v “cgroup success”
observer.log.20260204141806783:[2026-02-04 14:10:45.163420] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=13][errcode=-4122] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4122, timeout=1000000000)
observer.log.20260204141806783:[2026-02-04 14:10:45.163454] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=32][errcode=-4122]
observer.log.20260204141806783:[2026-02-04 14:10:45.163493] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770185445163483, ref_timestamp=1770184445702862, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204141806783:[2026-02-04 14:10:45.163512] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=15][errcode=0] long time to hold one guard(schema_mgr=0x7eee964194e0, tenant_id=1002, version=1770184445593264, cur_timestamp=1770185445163508, ref_timestamp=1770184445702862, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204143343380:[2026-02-04 14:27:25.179998] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=7][errcode=-4122] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4122, timeout=1000000000)
observer.log.20260204143343380:[2026-02-04 14:27:25.180024] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=23][errcode=-4122]
observer.log.20260204145019401:[2026-02-04 14:44:05.137644] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=7][errcode=-4012] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4012, timeout=1000000000)
observer.log.20260204145019401:[2026-02-04 14:44:05.137672] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=26][errcode=-4012]
observer.log.20260204145019401:[2026-02-04 14:44:05.137709] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=13][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770187445137700, ref_timestamp=1770186445208695, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204145019401:[2026-02-04 14:44:05.137734] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=20][errcode=0] long time to hold one guard(schema_mgr=0x7eee964194e0, tenant_id=1002, version=1770184445593264, cur_timestamp=1770187445137729, ref_timestamp=1770186445208695, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204145019401:[2026-02-04 14:46:35.707282] INFO [SERVER] cancel_task (ob_sys_task_stat.cpp:280) [1289851][T1_L0_G0][T1][YB420A8C0619-000649E222EF5AC9-0-0] [lt=22] cancel task(task={start_time:1770135258824876, task_id:Y100000ADF6316-000649EDBB42C88B-0-0, task_type:2, svr_ip:“xx.xx.xx.22:2882”, tenant_id:1002, is_cancel:true, comment:""}, i=13)

observer.log.20260204150707194:[2026-02-04 15:00:45.212949] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=34][errcode=-4012]
observer.log.20260204150707194:[2026-02-04 15:00:45.213006] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth]
observer.log.20260204152348748:[2026-02-04 15:17:25.241441] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=1][errcode=-4122] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4122, timeout=1000000000)
observer.log.20260204152348748:[2026-02-04 15:17:25.241470] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=27][errcode=-4122]
observer.log.20260204152348748:[2026-02-04 15:17:25.241517] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770189445241507, ref_timestamp=1770188445363091, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204152348748:[2026-02-04 15:17:25.241538] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=0] long time to hold one guard(schema_mgr=0x7eee964194e0, tenant_id=1002, version=1770184445593264, cur_timestamp=1770189445241534, ref_timestamp=1770188445363091, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204155658234:[2026-02-04 15:50:45.201228] WDIAG [RPC] send (ob_poc_rpc_proxy.h:171) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4012] sync rpc execute fail(ret=-4012, addr=“xx.xx.xx.22:2882”, pcode=5144, timeout=1000000000)
observer.log.20260204155658234:[2026-02-04 15:50:45.201291] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=14][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770191445201282, ref_timestamp=1770190445197708, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204155658234:[2026-02-04 15:50:45.201312] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=0] long time to hold one guard(schema_mgr=0x7eee964194e0, tenant_id=1002, version=1770184445593264, cur_timestamp=1770191445201307, ref_timestamp=1770190445197708, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)

#10

observer.log.20260204150707194:[2026-02-04 15:00:45.212949] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=34][errcode=-4012]
observer.log.20260204150707194:[2026-02-04 15:00:45.213006] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth]
observer.log.20260204152348748:[2026-02-04 15:17:25.241441] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=1][errcode=-4122] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4122, timeout=1000000000)
observer.log.20260204152348748:[2026-02-04 15:17:25.241470] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=27][errcode=-4122]
observer.log.20260204152348748:[2026-02-04 15:17:25.241517] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770189445241507, ref_timestamp=1770188445363091, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204152348748:[2026-02-04 15:17:25.241538] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289705][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=0] long time to hold one guard(schema_mgr=0x7eee964194e0, tenant_id=1002, version=1770184445593264, cur_timestamp=1770189445241534, ref_timestamp=1770188445363091, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204155658234:[2026-02-04 15:50:45.201228] WDIAG [RPC] send (ob_poc_rpc_proxy.h:171) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4012] sync rpc execute fail(ret=-4012, addr=“xx.xx.xx.22:2882”, pcode=5144, timeout=1000000000)
observer.log.20260204155658234:[2026-02-04 15:50:45.201291] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=14][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770191445201282, ref_timestamp=1770190445197708, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204155658234:[2026-02-04 15:50:45.201312] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289703][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=0] long time to hold one guard(schema_mgr=0x7eee964194e0, tenant_id=1002, version=1770184445593264, cur_timestamp=1770191445201307, ref_timestamp=1770190445197708, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
[apps@gd16c-db-115022-db-ob log]$ grep ‘Y100000ADF6316-000649EDBB42C88B-0-0’ observer.log.2026020416*|grep -v “cgroup success”
observer.log.20260204161329364:[2026-02-04 16:07:25.250381] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289704][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=8][errcode=-4012] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4012, timeout=1000000000)
observer.log.20260204161329364:[2026-02-04 16:07:25.250470] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289704][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=87][errcode=-4012]
observer.log.20260204161329364:[2026-02-04 16:07:25.250653] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289704][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770192445250500, ref_timestamp=1770191445330252, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204161329364:[2026-02-04 16:07:25.250690] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289704][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=0] long time to hold one guard(schema_mgr=0x7eee964194e0, tenant_id=1002, version=1770184445593264, cur_timestamp=1770192445250673, ref_timestamp=1770191445330252, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204163009917:[2026-02-04 16:24:05.313163] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4122] execute rpc fail(addr=“xx.xx.xx.22:2882”, pcode=5144, ret=-4122, timeout=1000000000)
observer.log.20260204163009917:[2026-02-04 16:24:05.313192] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:247) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=27][errcode=-4122]
observer.log.20260204163009917:[2026-02-04 16:24:05.313234] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770193445313224, ref_timestamp=1770192445493254, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204163009917:[2026-02-04 16:24:05.313267] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=26][errcode=0] long time to hold one guard(schema_mgr=0x7eee9642e9f0, tenant_id=1002, version=1770192445299128, cur_timestamp=1770193445313261, ref_timestamp=1770192445493254, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)

#11

mysql> SELECT task_id, object_id, target_object_id, ret_code FROM oceanbase.__all_virtual_ddl_task_status WHERE status = 99 AND tenant_id = 1002\G;
*************************** 1. row ***************************
task_id: 694268041
object_id: 508104
target_object_id: 578205
ret_code: -4072
*************************** 2. row ***************************
task_id: 694268145
object_id: 506348
target_object_id: 578206
ret_code: -4072
*************************** 3. row ***************************
task_id: 694269593
object_id: 508312
target_object_id: 578208
ret_code: -4072
*************************** 4. row ***************************
task_id: 694271047
object_id: 508336
target_object_id: 578213
ret_code: -4072
*************************** 5. row ***************************
task_id: 694271182
object_id: 508226
target_object_id: 578214
ret_code: -4072
*************************** 6. row ***************************
task_id: 694362512
object_id: 509552
target_object_id: 578220
ret_code: -4072
*************************** 7. row ***************************
task_id: 694363935
object_id: 509449
target_object_id: 578222
ret_code: -4072
*************************** 8. row ***************************
task_id: 694424141
object_id: 509841
target_object_id: 578223
ret_code: -4072
*************************** 9. row ***************************
task_id: 694439037
object_id: 509207
target_object_id: 578224
ret_code: -4072
*************************** 10. row ***************************
task_id: 694441780
object_id: 509291
target_object_id: 578225
ret_code: -4072
*************************** 11. row ***************************
task_id: 694450029
object_id: 509167
target_object_id: 578227
ret_code: -4072
*************************** 12. row ***************************
task_id: 694451392
object_id: 507325
target_object_id: 578229
ret_code: -4072
*************************** 13. row ***************************
task_id: 694452557
object_id: 507001
target_object_id: 578230
ret_code: -4072
*************************** 14. row ***************************
task_id: 734956301
object_id: 506840
target_object_id: 579191
ret_code: -4072
14 rows in set (0.01 sec)

message太长了
018E808080000120828A8C80400001000000000001C7AC808000018C808080000000EA0700000000000000000001FDA7808000EA0791EB1EFFFFFFFFFFFFFFFFFF01F7AC232C01000000808001000010D5FD96BBB8ECA6250000030001022E0001FFFFFFFF0F0190F3ACDADBBD92030001868080800000000001000001868080800000000000000000000000225F5F68696464656E5F3537393139315F3530363834307461626C655F736368656D610006000001918080800000000000000000FFFFFFFFFFFFFFFFFF010000001D01818180800001E880808000EA07F7AC231088DABC80CCA19203010000000A053FFFFFFFFFFFCF020000010001000000063FFF00063FFF026964000CE887AAE5A29EE5BA8FE58FB7000000001100FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001828180800001E980808000EA07F7AC231188DABC80CCA19203000000000A053FFFFFFFFFFFCF020000000001000000063FFF00063FFF07757365725F69640008E794A8E688B76964000000101200FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001838180800001EA80808000EA07F7AC231288DABC80CCA19203000000000A053FFFFFFFFFFFCF020000000001000000063FFF00063FFF086F726465725F69640008E8AEA2E58D956964000000111300FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001C58180800001AC81808000EA07F7AC231388DABC80CCA192030000000006053FFFFFFFFFFF3F0000000001000000063FFF00063FFF04666C6167004FE99C80E8A681E5908CE6ADA5E79A84E8AEA2E58D95E78AB6E6808133E4B8BAE696B0E5BBBA2C3233E4B8BAE58F96E6B6882C3235E4B8BAE694B9E8A1A8E5A4B42C3234E8AEA2E58D95E4BF9DE79599000000121400FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001AE81808000019581808000EA07F7AC231488DABC80CCA192030000000017072EFF8E808080F0FFFFFFFF010000000002000000063FFF17022EFF0000086F726465725F736E002DE5B9B6E99D9EE5BF85E5A1ABEFBC8CE698AFE4B8BAE4BA86E4BFAEE694B9E8A1A8E8BAABE8808CE5819AE79A84000000131500FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001B681808000019D81808000EA07F7AC231588DABC80CCA192030000000017072EFF8E808080F0FFFFFFFF010000000002000000063FFF00063FFF0C6F6C645F6F726465725F736E0033E697A7E79A84E8AEA2E58D95E58FB7E794A8E4BD9CE4BFAEE694B9E8A1A8E8BAABE5908EE697A7E8AEA2E58D95E585B3E88194000000141600FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001AC81808000019381808000EA07F7AC231688DABC80CCA192030000000006053FFFFFFFFFFF3F0000000001000000063FFF06053F000108776D735F666C61670031574D53E78AB6E68081E5908CE6ADA5E6A087E8AEB02C31E4B8BAE696B0E7BAAAE5BD952C30E4B8BAE5B7B2E5908CE6ADA5000000151700FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001CD8180800001B481808000EA07F7AC231788DABC80CCA192030000000006053FFFFFFFFFFF3F0000000001000000063FFF00063FFF0D616C6C6F636174655F666C6167004EE5BA93E5AD98E58886E9858DE6A087E5BF972C303AE69CAAE58886E9858D2C313AE5B7B2E58886E9858D2C323AE58886E9858DE5A4B1E8B4A52C333AE697A0E99C80E5BA93E5AD98E58886E9858D000000161800FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC231888DABC80CCA192030000000016072EFF8A808080F0FFFFFFFF010000000002000000063FFF16022EFF00000977617265686F7573650009E58F91E8B4A7E4BB93000000171900FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC231988DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726131000CE689A9E5B195E5AD97E6AEB5000000181A00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC231A88DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726132000CE689A9E5B195E5AD97E6AEB5000000191B00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC231B88DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726133000CE689A9E5B195E5AD97E6AEB50000001A1C00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC231C88DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726134000CE689A9E5B195E5AD97E6AEB50000001B1D00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC231D88DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726135000CE689A9E5B195E5AD97E6AEB50000001C1E00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC231E88DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726136000CE689A9E5B195E5AD97E6AEB50000001D1F00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC231F88DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726137000CE689A9E5B195E5AD97E6AEB50000001E2000FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC232088DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726138000CE689A9E5B195E5AD97E6AEB50000001F2100FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC232188DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF000006657874726139000CE689A9E5B195E5AD97E6AEB5000000202200FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018C8180800001F380808000EA07F7AC232288DABC80CCA192030000000016072EFFB2808080F0FFFFFFFF010000000002000000063FFF16022EFF00000765787472613130000CE689A9E5B195E5AD97E6AEB5000000212300FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC232388DABC80CCA192030000000012053FFFFFFFFFFFBF020000000001000000063FFF19070000140B6372656174655F74696D65000CE5889BE5BBBAE697B6E997B4000000222400FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC232488DABC80CCA192030000000004053FFFFFFFFFFFBF010000000001000000063FFF04053F00000B72657472795F636F756E74000CE9878DE8AF95E6ACA1E695B0000000232500FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000019E81808000018581808000EA07F7AC232588DABC80CCA192030000000016072EFF94808080F0FFFFFFFF010000000002000000063FFF16022EFF0130000C646567726164655F747970650018E8AEB0E5BD95E8AEA2E58D95E9998DE7BAA7E4BFA1E681AF000000242600FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000018B8180800001F280808000EA07F7AC232688DABC80CCA192030000000012053FFFFFFFFFFFBF020000000001000100063FFF19070000140B7570646174655F74696D65000CE69BB4E696B0E697B6E997B4000000252700FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000019981808000018081808000EA07F7AC232788DABC80CCA192030000000012053FFFFFFFFFFFBF020000000001000000063FFF12053F0080C0E1FEAFDE470D616C6C6F636174655F74696D650012E58886E9858DE58DA0E794A8E697B6E997B4000000262800FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001918180800001F880808000EA07F7AC232888DABC80CCA192030000000012053FFFFFFFFFFFBF020000000001000000063FFF12053F0080C0E1FEAFDE470B6578706F72745F74696D65000CE8BE93E587BAE697B6E997B4000000272900FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001958180800001FC80808000EA07F7AC232988DABC80CCA192030000000012053FFFFFFFFFFFBF020000000001000000063FFF12053F0080C0E1FEAFDE47096E6578745F74696D650012E4B88BE6ACA1E689A7E8A18CE697B6E997B4000000282A00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001DD8180800001C481808000EA07F7AC232A88DABC80CCA192030000000006053FFFFFFFFFFF3F0000000001000000063FFF06053F00000970726F635F737465700061303A20E9BB98E8AEA4E580BC28E58E86E58FB2E695B0E68DAE292C20313A20E4BB93E4B8ADE4BB93E5AFBBE4BB932C20323A20E5AFBB6A6974782C20333A20E5AE9EE789A9E5BA93E5AD98E58886E9858D20342E20E8AEA2E58D95E8BE93E587BA000000292B00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF010000018180808000008080100181808080000000000000000000000000000000000000000000019C81808000018381808000EA07F7AC232B88DABC80CCA19203000000000A053FFFFFFFFFFFCF020000000001000000063FFF0A053F00000D64656C69766572795F74696D65001BE58F91E8B4A7E697B6E997B4EFBC8CE58D95E4BD8DEFBC9AE7A7920000002A2C00FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000001938180800001FA80808000EA07F7AC232C88DABC80CCA19203000000000A053FFFFFFFFFFFCF020000000001000000063FFF0A053F00000D6578745F62697A5F666C6167730012E689A9E5B195E4B89AE58AA1E6A087E8AF860000002B0000FFFFFFFFFFFFFFFF7FE0FFFFFFFFFFFFFFFF01000001818080800000808010018180808000000000000000000000000000000000000000000000000000008080804000000000000000000000000100000080600000FFFFFFFFFFFFFFFFFF01000000FFFFFFFFFFFFFFFFFF0109018580808000DEF71E0501018580808000F2F71E050101858080800081F81E05010185808080008CF81E050101858080800094F81E05010185808080009BF81E0501018580808000A1F81E0501018580808000A4F81E0501018580808000B5F81E05010001FFFFFFFFFFFFFFFFFF01D8F71EFFFFFFFFFFFFFFFFFF010000000000000003761EFFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF0100000000000100FFFFFFFFFFFFFFFFFF0100000000000000000000000000804002019E80808000010C5F5F636F5F64656661756C74000088DABC80CCA1920380800106010001B880808000E907085F5F636F5F616C6C000188DABC80CCA1920380800106011D101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C01E907FFFFFFFFFFFFFFFFFF0100000001818080800000000000000000000004117B22474C4F42414C223A224155544F227D000000000000106F66635F73796E635F646174615F323100167669705F6F66635F73686172645F617263686976657300167669705F6F66635F73686172645F617263686976657300FFFFFFFFFFFFFFFFFF010180C020A080804000000000018980808000FFFFFFFF0F80E10100E807E807DFABD8800C019280808000020000018980808000FFFFFFFF0F80E101000944442D4D4F4E2D5252001844442D4D4F4E2D52522048482E4D492E535358464620414D001C44442D4D4F4E2D52522048482E4D492E535358464620414D20545A52000001B581808000018B808080000000000000000000000000FFFFFFFFFFFFFFFFFF0100019081808000FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF01000002D580808000018880808000C00000010100000001848080800080000000019080808000C2000003090000000463500D00D8F2320190808080003E000003090000000463500D01D8F232018880808000C00000011400000000000000FFFFFFFFFFFFFFFFFF010000000000000010C0000000FFFFFFFF0FFFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF0100000000000000000001000186808080000400000000000181808080000001A980808000FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF01FFFFFFFFFFFFFFFFFF0100020000000014000000000192808080000000000000000004000000000700FF00000000018E808080000000000000000700FF000000000000000001010101000000000000000000017FFFFFFFFFFFFFFF0001

#12

observer.log.20260204231138886:[2026-02-04 23:04:05.856765] WDIAG [RPC] send (ob_poc_rpc_proxy.h:171) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=7][errcode=-4012] sync rpc execute fail(ret=-4012, addr=“xx.xx.xx.22:2882”, pcode=5144, timeout=1000000000)
observer.log.20260204231138886:[2026-02-04 23:04:05.856824] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=15][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770217445856815, ref_timestamp=1770216445852422, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204231138886:[2026-02-04 23:04:05.856853] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289706][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=20][errcode=0] long time to hold one guard(schema_mgr=0x7eee96555f20, tenant_id=1002, version=1770203446134616, cur_timestamp=1770217445856845, ref_timestamp=1770216445852422, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204234514636:[2026-02-04 23:37:25.886961] WDIAG [RPC] send (ob_poc_rpc_proxy.h:171) [1289700][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=10][errcode=-4012] sync rpc execute fail(ret=-4012, addr=“xx.xx.xx.22:2882”, pcode=5144, timeout=1000000000)
observer.log.20260204234514636:[2026-02-04 23:37:25.887038] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289700][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770219445887012, ref_timestamp=1770218445881535, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260204234514636:[2026-02-04 23:37:25.887059] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289700][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=15][errcode=0] long time to hold one guard(schema_mgr=0x7eee96555f20, tenant_id=1002, version=1770203446134616, cur_timestamp=1770219445887054, ref_timestamp=1770218445881535, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)

持续在报这个，平均每10min报一次

#13

observer.log.20260205090418083:[2026-02-05 09:04:06.531566] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289702][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=15][errcode=0] long time to hold one guard(schema_mgr=0x7eee96555f20, tenant_id=1002, version=1770203446134616, cur_timestamp=1770253446531561, ref_timestamp=1770252446529587, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260205092110144:[2026-02-05 09:20:46.568998] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=7][errcode=-4012] REACH SYSLOG RATE LIMIT [bandwidth]
observer.log.20260205092110144:[2026-02-05 09:20:46.569043] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=1][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770254446569033, ref_timestamp=1770253446636813, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260205092110144:[2026-02-05 09:20:46.569064] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289699][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=17][errcode=0] long time to hold one guard(schema_mgr=0x7eee96555f20, tenant_id=1002, version=1770203446134616, cur_timestamp=1770254446569060, ref_timestamp=1770253446636813, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260205095433431:[2026-02-05 09:54:06.591878] WDIAG [RPC] send (ob_poc_rpc_proxy.h:171) [1289700][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=11][errcode=-4012] sync rpc execute fail(ret=-4012, addr=“xx.xx.xx.22:2882”, pcode=5144, timeout=1000000000)
observer.log.20260205095433431:[2026-02-05 09:54:06.591934] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289700][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7efc3e26f3d0, tenant_id=1, version=1769590007612728, cur_timestamp=1770256446591924, ref_timestamp=1770255446589586, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496daf 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)
observer.log.20260205095433431:[2026-02-05 09:54:06.591955] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1289700][T1_DDLTaskExecu][T1][Y100000ADF6316-000649EDBB42C88B-0-0] [lt=16][errcode=0] long time to hold one guard(schema_mgr=0x7eee96555f20, tenant_id=1002, version=1770203446134616, cur_timestamp=1770256446591950, ref_timestamp=1770255446589586, lbt()=“0x26a44e8d 0x9755615 0x9754389 0x97542b4 0x9496e83 0x11d6657a 0x11d67477 0x10ce341a 0x975b7bc 0x1002a10a 0x266b43d4 0x266bb307 0x266ba8c5 0x266b89d2 0x7efd752081ca 0x7efd74e39e73”)

现在还在报，ddl不是完全卡死，但是非常的慢，56个建表语句跑了3h+，中间还超时好几次

#14

image
image1515×381 30.6 KB

#15

SELECT table_id, table_name, table_type, data_table_id
FROM oceanbase.__all_virtual_table
WHERE tenant_id = 1002
AND table_id IN (578205, 578206, 578208, 578213, 578214, 578220, 578222, 578223, 578224, 578225, 578227, 578229, 578230, 579191);
查询下表还存在么

#16

±---------±-----------------------------------±-----------±--------------+
| table_id | table_name | table_type | data_table_id |
±---------±-----------------------------------±-----------±--------------+
| 578205 | __hidden_578205_508104table_schema | 3 | 0 |
| 578206 | __hidden_578206_506348table_schema | 3 | 0 |
| 578208 | __hidden_578208_508312table_schema | 3 | 0 |
| 578213 | __hidden_578213_508336table_schema | 3 | 0 |
| 578214 | __hidden_578214_508226table_schema | 3 | 0 |
| 578220 | __hidden_578220_509552table_schema | 3 | 0 |
| 578222 | __hidden_578222_509449table_schema | 3 | 0 |
| 578223 | __hidden_578223_509841table_schema | 3 | 0 |
| 578224 | __hidden_578224_509207table_schema | 3 | 0 |
| 578225 | __hidden_578225_509291table_schema | 3 | 0 |
| 578227 | __hidden_578227_509167table_schema | 3 | 0 |
| 578229 | __hidden_578229_507325table_schema | 3 | 0 |
| 578230 | __hidden_578230_507001table_schema | 3 | 0 |
| 579191 | __hidden_579191_506840table_schema | 3 | 0 |
±---------±-----------------------------------±-----------±--------------+
14 rows in set (0.01 sec)

#17

这些hidden表未删除可能是卡住了，查询下这些 hidden table 是否有锁

#18

image
image1251×564 100 KB

image
image1524×406 33.6 KB

迁移的rps已经掉0，貌似内存被hold住了

#19

ERROR 1146 (42S02): Table ‘oceanbase.__all_virtual_table_lock_with_tenant’ doesn’t exist
ERROR 1146 (42S02): Table ‘oceanbase.__all_virtual_table_lock_with_tenant’ doesn’t exist

用户租户，sys租户均没有这个表

#20

SELECT task_id, object_id, target_object_id, ret_code FROM oceanbase.__all_virtual_ddl_task_status WHERE status = 99 AND tenant_id = 1002\G; 根据这个查出来的task_id信息
查一下 这个信息 看看ddl执行 到哪一步了？
select * from __all_rootservice_event_history where module like ‘ddl_scheduler’ and value4 like ‘ddl_task_id’;

这个信息 也查一下
select id, svr_ip, svr_port, tenant, command, trace_id, time, total_time, retry_cnt, regexp_replace(replace(info,’\n’,’’), ’ +’, ’ ') as stmt from __all_virtual_processlist where command not like ‘Sleep’ order by stmt;

#21

– 检查 hidden table 和原表的 DDL 锁
SELECT * FROM oceanbase.__all_virtual_obj_lock
WHERE tenant_id = 1002
AND obj_id IN (578205, 578206, 578208, 578213, 578214, 578220, 578222, 578223, 578224, 578225, 578227, 578229, 578230, 579191,
508104, 506348, 508312, 508336, 508226, 509552, 509449, 509841, 509207, 509291, 509167, 507325, 507001, 506840);