由于删除大表，转了半天没动。尝试多少删除依旧无法删除。重启OB 租户就连接不上了
由于删除大表，转了半天没动。尝试次删除依旧无法删除。重启OB 租户就连接不上了
集群 sys 租户 还可以连吗？如果可以就查看集群事件日志
select * from __all_rootservice_event_History
order by gmt_create desc limit 50;
集群 sys 租户可以链接
没有这张表 __all_rootservice_event_Historyorder
只看了 __all_rootservice_event_history
就是这个。抱歉，手机上编辑，换行失效了。
重新看看。
ERROR 4072 (HY000) : Operation canceled
OceanBase 错误码：4072
错误原因：操作被取消。
解决方式：请联系技术支持人员协助排查。
前面几次 drop 都失败了。看最后好像成功了。
登录业务租户看看 drop 成功了么
有可能clog满了。
在observer.log里找找最后的几个error日志
租户无法启动。里面还有数据 
导致原因是删除大表，百万以上的表。有好几张表这种，
使用第一次Navicat 删除，等半天没有反应，我断开链接。再次发起删除没有反应，来回搞几次。
这个时候还能操作，只是表没有删除。。
后来重新启动集群，租户就无法链接使用了
详细如下：
- 集群参数和目录
-bash-4.2$ strings etc/observer.config.bin|grep dir
data_dir=/www/obdata
-bash-4.2$ df -h |grep www
/dev/sdb1 881G 450G 386G 54% /www
[ob@mysql_master_38 oceanbase]$ strings etc/observer.config.bin|grep dir
data_dir=/www/obdata
[ob@mysql_master_38 oceanbase]$ df -h |grep www
[ob@mysql_master_38 oceanbase]$ df -h |grep /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 16K 16G 1% /dev/shm
tmpfs 16G 715M 15G 5% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda4 203G 5.9G 197G 3% /
[ob@mysql_master_38 oceanbase]$ strings etc/observer.config.bin|grep dir
data_dir=/www/obdata
[ob@mysql_master_38 oceanbase]$ df -h |grep www
[ob@mysql_master_38 oceanbase]$ df -h |grep /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 16K 16G 1% /dev/shm
tmpfs 16G 715M 15G 5% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda4 203G 5.9G 197G 3% /
- 集群状态
mysql> select zone,concat(SVR_IP,':',SVR_PORT) observer,
-> cpu_capacity_max cpu_total,cpu_assigned_max cpu_assigned,
-> cpu_capacity-cpu_assigned_max as cpu_free,
-> round(memory_limit/1024/1024/1024,2) as memory_total,
-> round((memory_limit-mem_capacity)/1024/1024/1024,2) as system_memory,
-> round(mem_assigned/1024/1024/1024,2) as mem_assigned,
-> round((mem_capacity-mem_assigned)/1024/1024/1024,2) as memory_free,
-> round(log_disk_capacity/1024/1024/1024,2) as log_disk_capacity,
-> round(log_disk_assigned/1024/1024/1024,2) as log_disk_assigned,
-> round((log_disk_capacity-log_disk_assigned)/1024/1024/1024,2) as log_disk_free,
-> round((data_disk_capacity/1024/1024/1024),2) as data_disk,
-> round((data_disk_in_use/1024/1024/1024),2) as data_disk_used,
-> round((data_disk_capacity-data_disk_in_use)/1024/1024/1024,2) as data_disk_free
-> from gv$ob_servers;
+-------+-------------------+-----------+--------------+----------+--------------+---------------+--------------+-------------+-------------------+-------------------+---------------+-----------+----------------+----------------+
| zone | observer | cpu_total | cpu_assigned | cpu_free | memory_total | system_memory | mem_assigned | memory_free | log_disk_capacity | log_disk_assigned | log_disk_free | data_disk | data_disk_used | data_disk_free |
+-------+-------------------+-----------+--------------+----------+--------------+---------------+--------------+-------------+-------------------+-------------------+---------------+-----------+----------------+----------------+
| zone2 | 192.168.2.36:2882 | 50 | 32 | 18 | 24.00 | 5.00 | 16.00 | 3.00 | 74.00 | 45.00 | 29.00 | 107.00 | 9.45 | 97.55 |
| zone1 | 192.168.2.31:2882 | 50 | 32 | 18 | 24.00 | 5.00 | 16.00 | 3.00 | 74.00 | 45.00 | 29.00 | 107.00 | 9.52 | 97.48 |
| zone3 | 192.168.2.38:2882 | 50 | 32 | 18 | 24.00 | 5.00 | 16.00 | 3.00 | 74.00 | 45.00 | 29.00 | 107.00 | 9.75 | 97.25 |
+-------+-------------------+-----------+--------------+----------+--------------+---------------+--------------+-------------+-------------------+-------------------+---------------+-----------+----------------+----------------+
3 rows in set (0.00 sec)
- 租户资源
mysql> select t1.name resource_pool_name, t2.`name` unit_config_name,
-> t2.max_cpu, t2.min_cpu,
-> round(t2.memory_size/1024/1024/1024,2) mem_size_gb,
-> round(t2.log_disk_size/1024/1024/1024,2) log_disk_size_gb, t2.max_iops,
-> t2.min_iops, t3.unit_id, t3.zone, concat(t3.svr_ip,':',t3.`svr_port`) observer,
-> t4.tenant_id, t4.tenant_name
-> from __all_resource_pool t1
-> join __all_unit_config t2 on (t1.unit_config_id=t2.unit_config_id)
-> join __all_unit t3 on (t1.`resource_pool_id` = t3.`resource_pool_id`)
-> left join __all_tenant t4 on (t1.tenant_id=t4.tenant_id)
-> order by t1.`resource_pool_id`, t2.`unit_config_id`, t3.unit_id;
+--------------------------+-----------------------------------+---------+---------+-------------+------------------+----------+----------+---------+-------+-------------------+-----------+-------------+
| resource_pool_name | unit_config_name | max_cpu | min_cpu | mem_size_gb | log_disk_size_gb | max_iops | min_iops | unit_id | zone | observer | tenant_id | tenant_name |
+--------------------------+-----------------------------------+---------+---------+-------------+------------------+----------+----------+---------+-------+-------------------+-----------+-------------+
| sys_pool | sys_unit_config | 1 | 1 | 2.00 | 2.00 | 10000 | 10000 | 1 | zone1 | 192.168.2.31:2882 | 1 | sys |
| sys_pool | sys_unit_config | 1 | 1 | 2.00 | 2.00 | 10000 | 10000 | 2 | zone2 | 192.168.2.36:2882 | 1 | sys |
| sys_pool | sys_unit_config | 1 | 1 | 2.00 | 2.00 | 10000 | 10000 | 3 | zone3 | 192.168.2.38:2882 | 1 | sys |
| ocp_pool | ocp_unit | 1 | 1 | 2.00 | 7.00 | 10000 | 10000 | 1001 | zone1 | 192.168.2.31:2882 | 1002 | ocp |
| ocp_pool | ocp_unit | 1 | 1 | 2.00 | 7.00 | 10000 | 10000 | 1002 | zone2 | 192.168.2.36:2882 | 1002 | ocp |
| ocp_pool | ocp_unit | 1 | 1 | 2.00 | 7.00 | 10000 | 10000 | 1003 | zone3 | 192.168.2.38:2882 | 1002 | ocp |
| pool_zhongjing_zone3_jxr | config_zhongjing_zone3_30C12G_nuo | 30 | 30 | 12.00 | 36.00 | 300000 | 300000 | 1004 | zone3 | 192.168.2.38:2882 | 1004 | zhongjing |
| pool_zhongjing_zone1_wnv | config_zhongjing_zone1_30C12G_bqc | 30 | 30 | 12.00 | 36.00 | 300000 | 300000 | 1005 | zone1 | 192.168.2.31:2882 | 1004 | zhongjing |
| pool_zhongjing_zone2_fbg | config_zhongjing_zone2_30C12G_fus | 30 | 30 | 12.00 | 36.00 | 300000 | 300000 | 1006 | zone2 | 192.168.2.36:2882 | 1004 | zhongjing |
+--------------------------+-----------------------------------+---------+---------+-------------+------------------+----------+----------+---------+-------+-------------------+-----------+-------------+
9 rows in set (0.00 sec)
- 租户资源使用情况
mysql> select a.svr_ip,a.svr_port,a.tenant_id,b.tenant_name,
-> CAST(a.data_disk_in_use/1024/1024/1024 as DECIMAL(15,2)) data_disk_use_G,
-> CAST(a.log_disk_size/1024/1024/1024 as DECIMAL(15,2)) log_disk_size,
-> CAST(a.log_disk_in_use/1024/1024/1024 as DECIMAL(15,2)) log_disk_use_G
-> from __all_virtual_unit a,dba_ob_tenants b
-> where a.tenant_id=b.tenant_id;
+--------------+----------+-----------+-------------+-----------------+---------------+----------------+
| svr_ip | svr_port | tenant_id | tenant_name | data_disk_use_G | log_disk_size | log_disk_use_G |
+--------------+----------+-----------+-------------+-----------------+---------------+----------------+
| 192.168.2.31 | 2882 | 1 | sys | 0.11 | 2.00 | 0.81 |
| 192.168.2.31 | 2882 | 1001 | META$1002 | 0.27 | 0.70 | 0.54 |
| 192.168.2.31 | 2882 | 1002 | ocp | 0.15 | 6.30 | 1.82 |
| 192.168.2.31 | 2882 | 1003 | META$1004 | 0.13 | 3.60 | 0.58 |
| 192.168.2.31 | 2882 | 1004 | zhongjing | 8.77 | 32.40 | 25.87 |
| 192.168.2.38 | 2882 | 1 | sys | 0.10 | 2.00 | 0.81 |
| 192.168.2.38 | 2882 | 1001 | META$1002 | 0.27 | 0.70 | 0.54 |
| 192.168.2.38 | 2882 | 1002 | ocp | 0.16 | 6.30 | 1.82 |
| 192.168.2.38 | 2882 | 1003 | META$1004 | 0.12 | 3.60 | 0.58 |
| 192.168.2.38 | 2882 | 1004 | zhongjing | 8.99 | 32.40 | 25.87 |
| 192.168.2.36 | 2882 | 1 | sys | 0.10 | 2.00 | 0.81 |
| 192.168.2.36 | 2882 | 1001 | META$1002 | 0.25 | 0.70 | 0.54 |
| 192.168.2.36 | 2882 | 1002 | ocp | 0.15 | 6.30 | 1.82 |
| 192.168.2.36 | 2882 | 1003 | META$1004 | 0.13 | 3.60 | 0.58 |
| 192.168.2.36 | 2882 | 1004 | zhongjing | 8.71 | 32.40 | 25.87 |
+--------------+----------+-----------+-------------+-----------------+---------------+----------------+
15 rows in set (0.02 sec)
- 集群事件
mysql> select gmt_create , module , event, name1 , value1 , name2, value2, name3, value3 , name4, value4, rs_svr_ip from `__all_rootservice_event_history` areh WHERE 1=1 and module in ('inspector') order by gmt_create desc limit 50;
;
+----------------------------+-----------+----------------+-------+----------------------------+-----------+--------+-------------+-----------+-------+--------+--------------+
| gmt_create | module | event | name1 | value1 | name2 | value2 | name3 | value3 | name4 | value4 | rs_svr_ip |
+----------------------------+-----------+----------------+-------+----------------------------+-----------+--------+-------------+-----------+-------+--------+--------------+
| 2023-07-30 17:33:16.689315 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 17:23:16.696941 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 17:13:16.720380 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 16:53:16.707005 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 16:43:16.695955 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 16:33:16.695132 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 16:23:16.702886 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 16:13:16.693379 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 15:53:16.695980 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 15:43:16.703068 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 15:33:16.691528 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 15:23:16.697069 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 15:13:16.703018 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 15:03:16.724510 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 14:43:16.709270 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 14:33:16.702649 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 14:23:16.702348 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 14:13:16.715567 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 14:03:16.704032 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 13:53:16.738872 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 13:43:16.703645 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 13:33:16.886478 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 13:22:24.859281 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 13:12:24.795293 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 13:02:24.802519 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 12:42:24.837876 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 12:32:24.841124 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 12:22:24.850756 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 12:12:24.821512 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 12:02:24.828829 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 11:52:24.885350 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
| 2023-07-30 11:32:24.918770 | inspector | tenant_checker | info | tenant maybe create failed | tenant_id | 1004 | tenant_name | zhongjing | | | 192.168.2.31 |
+----------------------------+-----------+----------------+-------+----------------------------+-----------+--------+-------------+-----------+-------+--------+--------------+
32 rows in set (0.01 sec)
- 租户登录异常
[root@mysql_master_38 ~]# mysql -h121.201.2.38 -uroot@zhongjing -P2881 -p'!@#ZhongJing123' -c -A oceanbase
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 8001 (08004): Server is initializing
[root@mysql_master_38 ~]# mysql -h121.201.2.31 -uroot@zhongjing -P2881 -p'!@#ZhongJing123' -c -A oceanbase
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 8001 (08004): Server is initializing
[root@mysql_master_38 ~]# mysql -h121.201.2.36 -uroot@zhongjing -P2881 -p'!@#ZhongJing123' -c -A oceanbase
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 8001 (08004): Server is initializing
- 集群节点日志
节点 38
[ob@mysql_master_38 log]$ grep ERROR *.log*
observer.log:[2023-07-30 17:55:22.996369] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:2137) [73976][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4014] user tenant has been dropped(ret=-4014, ret="OB_INNER_STAT_ERROR", tenant_id=1)
observer.log:[2023-07-30 17:55:23.145750] INFO [SHARE.SCHEMA] init_sys_table_name_map (ob_schema_struct.cpp:225) [73976][][T0][Y0-0000000000000000-0-0] [lt=6] set tenant space table name(key=14220930887511194059, table={database_id:201006, name_case_mode:2, table_name:"ALL_VIRTUAL_TENANT_ERROR_REAL_AGENT"}, strlen=35)
observer.log.20230730132251509:[2023-07-30 13:22:42.350161] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:2137) [67917][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4014] user tenant has been dropped(ret=-4014, ret="OB_INNER_STAT_ERROR", tenant_id=1)
observer.log.20230730132251509:[2023-07-30 13:22:42.502547] INFO [SHARE.SCHEMA] init_sys_table_name_map (ob_schema_struct.cpp:225) [67917][][T0][Y0-0000000000000000-0-0] [lt=6] set tenant space table name(key=14220930887511194059, table={database_id:201006, name_case_mode:2, table_name:"ALL_VIRTUAL_TENANT_ERROR_REAL_AGENT"}, strlen=35)
rootservice.log:[2023-07-30 11:11:49.417542] WDIAG [RS] get_frozen_status (ob_major_freeze_helper.cpp:416) [62190][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4121] fail to get freeze info(ret=-4121, ret="OB_RPC_SEND_ERROR", frozen_scn={val:0}, tenant_id=1001)
rootservice.log:[2023-07-30 11:11:49.417556] WDIAG [RS] get_frozen_scn (ob_major_freeze_helper.cpp:431) [62190][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4121] fail to get frozen info(ret=-4121, ret="OB_RPC_SEND_ERROR")
rootservice.log:[2023-07-30 11:11:49.417685] WDIAG [RS] refresh_tenant_info (ob_tenant_info_loader.cpp:215) [62339][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4121] failed to load tenant info(ret=-4121, ret="OB_RPC_SEND_ERROR", tenant_id=1002)
rootservice.log:[2023-07-30 11:11:49.417703] WDIAG [RS] run2 (ob_tenant_info_loader.cpp:130) [62339][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=-4121] failed to update tenant info(ret=-4121, ret="OB_RPC_SEND_ERRO ", tenant_id=1002, sql_proxy_=0x5597c6003dc8)
rootservice.log:[2023-07-30 11:11:49.417822] WDIAG [RS] get_frozen_status (ob_major_freeze_helper.cpp:416) [62346][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4121] fail to get freeze info(ret=-4121, ret="OB_RPC_SEND_ERROR", frozen_scn={val:0}, tenant_id=1002)
rootservice.log:[2023-07-30 11:11:49.417837] WDIAG [RS] get_frozen_scn (ob_major_freeze_helper.cpp:431) [62346][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4121] fail to get frozen info(ret=-4121, ret="OB_RPC_SEND_ERROR")
trace.log.20230730124658196:[2023-07-30 11:09:36.066898] [62131][T1_L0_G0][T1002][YB42C0A80226-000601ABAD50FAE5-0-0] {"trace_id":"000601ab-a4c7-b6c6-fc30-ebb8034b2916","name":"sql_compile","id":"000601ab-a4c7-b6d9-7961-08fb3608177d","start_ts":1690686575851225,"end_ts":1690686575900455,"parent_id":"000601ab-a4c7-b6ca-2ed2-512fedc7df7a","is_follow":false,"tags":[{"sql_id":"98F8E1830AFEA651EFE0E2A3A969973B"},{"database_id":201001},{"plan_hash":8814062241771393325},{"hit_plan":false},{"sql_text":"select /*+ MONITOR_AGENT QUERY_TIMEOUT(100000000) */ count(*) as cnt from CDB_INDEXES where status in ('ERROR','UNUSABLE')"}]}
trace.log.20230730124658196:[2023-07-30 11:12:28.376874] [64532][T1_L0_G0][T1003][YB42C0A80226-000601ABB8320F27-0-0] {"trace_id":"000601ab-af0c-c696-998f-be86cb5c6bce","name":"sql_compile","id":"000601ab-af0c-c6b1-e28a-3205fe95ff3c","start_ts":1690686748149425,"end_ts":1690686748186642,"parent_id":"000601ab-af0c-c69e-de7c-3d19ff6c7b04","is_follow":false,"tags":[{"sql_id":"98F8E1830AFEA651EFE0E2A3A969973B"},{"database_id":201001},{"plan_hash":8814062241771393325},{"hit_plan":false},{"sql_text":"select /*+ MONITOR_AGENT QUERY_TIMEOUT(100000000) */ count(*) as cnt from CDB_INDEXES where status in ('ERROR','UNUSABLE')"}]}
节点 31
-bash-4.2$ grep ERROR *.log*
observer.log.20230730131509404:[2023-07-30 13:12:24.795219] ERROR issue_dba_error (ob_log.cpp:1792) [227269][RSInspectTask][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_root_inspection.cpp", line_no=216, info="tenant maybe create failed")
observer.log.20230730174008674:[2023-07-30 13:22:46.689828] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:2137) [248833][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4014] user tenant has been dropped(ret=-4014, ret="OB_INNER_STAT_ERROR", tenant_id=1)
节点 36
-bash-4.2$ grep ERROR *.log*
observer.log.20230730131509404:[2023-07-30 13:12:24.795219] ERROR issue_dba_error (ob_log.cpp:1792) [227269][RSInspectTask][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file="ob_root_inspection.cpp", line_no=216, info="tenant maybe create failed")
observer.log.20230730174008674:[2023-07-30 13:22:46.689828] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:2137) [248833][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4014] user tenant has been dropped(ret=-4014, ret="OB_INNER_STAT_ERROR", tenant_id=1)