减少租户的unit个数失败，回滚后，租户的状态一组是修改中

OceanBase 技术问题
#17

image
image1408×109 4.26 KB

image
image1358×536 50.7 KB

#18

image
image1437×174 12.9 KB

image

#20

image
image1686×704 196 KB

#21

麻烦提供一份ocp-server日志.
/home/admin/ocp/log中，ocp-server.log

#22

已重启，但是还是没有解决

#23

你是先做的租户缩容 才发现合并失败么？[OBServer 合并失败] - #34，来自 AntTech_TLWCTG

#25

合并的失败的问题，一直都有。

#26

有刷新过ocp页面么。看看刷新后还不同么

#27

刷新了，还是那个问题

#28

image
image1663×391 39.8 KB

如何停止Root Service 任务执行呢，现在这个租户处于修改中，其他操作都操作不了了，修改unit个数也不行。

#29

是否还有记录到修改unit报错的截图？
OCP 缩减 unit 任务回退不会让内核任务回退。需要等待内核执行完。
所以实际上卡在了你job任务那里了

#30

应该就是这样了。

image
image1582×902 66.9 KB

#31

这里报错显示的是 alter unit_num没有结束。那时候的ob日志应该已经没了吧。
这边先去咨询下看看有没有什么应急方法
麻烦你去查询gv$ob_sql_audit视图查下sql的trace id

#32

image
image1351×705 47.8 KB

#33

模糊查询下
select * from gv$ob_sql_audit where QUERY_SQL like ‘%alter%’;
根据sql判断出当前卡在哪条sql上，使用该trace id 版取一份日志
参考指令：cat observer.log* |grep ‘trace id’ > xxxx.log

#34

image
image1368×580 75.6 KB

看不出来卡在哪条sql上了

#36

你这里显示正在alter，通过query sql字段找一下 alter 语句。

image
image1663×391 39.8 KB

#37

麻烦执行下如下查询

SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE='ALTER_RESOURCE_TENANT_UNIT_NUM' AND TENANT_ID=1002;

SELECT * FROM __all_virtual_balance_job\G;

SELECT * FROM  cdb_ob_transfer_tasks where tenant_id=1002\G;
#38

image
image1321×197 5.94 KB

image
image1279×431 8.83 KB

image
image1364×332 7.51 KB

#39

日志中有大量IO Error，大概率磁盘有问题，建议检查下，例如dmesg -T等

grep “IO error” observer.log|wc -l
601
grep “IO error” rootservice.log|wc -l
197

[2025-03-17 14:12:00.842352] WDIAG [COMMON] wait (ob_io_define.cpp:1885) [117882][T1_L5_G0][T1][YB420A0B0940-00063023DBCBF8EF-0-0] [lt=24][errcode=-4016] IO error, (ret=-4016, *result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:4096, offset_:1594640, size_:295, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4016, fs_errno_:0}, tenant_id_:1, tenant_io_mgr_:{ptr:0x7f7b311e4030}, user_data_buf_:null, buf_:null, io_callback_:0x7f78bb40e080, time_log:{begin_ts:1742191920821378, enqueue_used:2, dequeue_used:18, submit_used:1, return_used:20759, callback_enqueue_used:3, callback_dequeue_used:12, callback_finish_used:202, end_used:-56}})
[2025-03-17 14:12:00.845803] WDIAG [COMMON] wait (ob_io_define.cpp:1885) [117882][T1_L5_G0][T1][YB420A0B0940-00063023DBCBF8EF-0-0] [lt=13][errcode=-4016] IO error, (ret=-4016, *result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:4096, offset_:1594640, size_:295, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4016, fs_errno_:0}, tenant_id_:1, tenant_io_mgr_:{ptr:0x7f7b311e4030}, user_data_buf_:null, buf_:null, io_callback_:0x7f78bb40e080, time_log:{begin_ts:1742191920842461, enqueue_used:2, dequeue_used:16, submit_used:1, return_used:3144, callback_enqueue_used:2, callback_dequeue_used:13, callback_finish_used:-1, end_used:1742191920845789}})
[2025-03-17 14:12:01.037301] WDIAG [COMMON] wait (ob_io_define.cpp:1885) [3539440][T1003_DBMSSched][T1003][YB420A0B093E-00062FB95BF73DA3-0-0] [lt=38][errcode=-4016] IO error, (ret=-4016, *result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:4096, offset_:1581402, size_:1611, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4016, fs_errno_:0}, tenant_id_:1003, tenant_io_mgr_:{ptr:0x7f79ab3f4030}, user_data_buf_:null, buf_:null, io_callback_:0x7f7aa4274080, time_log:{begin_ts:1742191921036717, enqueue_used:3, dequeue_used:35, submit_used:1, return_used:391, callback_enqueue_used:3, callback_dequeue_used:18, callback_finish_used:-1, end_used:1742191921037287}})

[2025-03-17 14:03:20.062144] WDIAG [COMMON] wait (ob_io_define.cpp:1885) [117531][RootBalance][T1][YB420A0B093E-00062FB906EBB704-0-0] [lt=16][errcode=-4016] IO error, (ret=-4016, *result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:4096, offset_:1594640, size_:295, timeout_us_:10000000, result_ref_cnt_:1, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4016, fs_errno_:0}, tenant_id_:1, tenant_io_mgr_:{ptr:0x7f7b311e4030}, user_data_buf_:null, buf_:null, io_callback_:0x7f7ac6b6c080, time_log:{begin_ts:1742191400061597, enqueue_used:2, dequeue_used:35, submit_used:1, return_used:319, callback_enqueue_used:4, callback_dequeue_used:16, callback_finish_used:157, end_used:-27}})
[2025-03-17 14:03:20.062656] WDIAG [COMMON] wait (ob_io_define.cpp:1885) [117531][RootBalance][T1][YB420A0B093E-00062FB906EBB704-0-0] [lt=8][errcode=-4016] IO error, (ret=-4016, *result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:4096, offset_:1594640, size_:295, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4016, fs_errno_:0}, tenant_id_:1, tenant_io_mgr_:{ptr:0x7f7b311e4030}, user_data_buf_:null, buf_:null, io_callback_:0x7f7ac6b6c080, time_log:{begin_ts:1742191400062184, enqueue_used:1, dequeue_used:31, submit_used:1, return_used:298, callback_enqueue_used:2, callback_dequeue_used:8, callback_finish_used:-1, end_used:1742191400062636}})
[2025-03-17 14:03:20.067182] WDIAG [COMMON] wait (ob_io_define.cpp:1885) [117531][RootBalance][T1][YB420A0B093E-00062FB906EBB704-0-0] [lt=7][errcode=-4016] IO error, (ret=-4016, *result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:4096, offset_:1594640, size_:295, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4016, fs_errno_:0}, tenant_id_:1, tenant_io_mgr_:{ptr:0x7f7b311e4030}, user_data_buf_:null, buf_:null, io_callback_:0x7f7a777d0080, time_log:{begin_ts:1742191400066703, enqueue_used:1, dequeue_used:16, submit_used:1, return_used:341, callback_enqueue_used:2, callback_dequeue_used:8, callback_finish_used:133, end_used:-32}})