创建租户失败：ERROR 4009 (58030): IO error

OceanBase 技术问题
#7 
[test2024@obrd.32c.vd2-s2c0-n0 /home/test2024]
$cat /etc/sysctl.conf 
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
#8

/etc/sysctl.conf 文件下面全是注释，没有内容

#9

加一下试试看。
（可选）配置 sysctl.conf
和这个echo fs.aio-max-nr = 131072 >> /etc/sysctl.conf

#10

我权限不够，没有办法写配置文件，之前也没有配置过这个文件，可以创建租户的来着

#11

老师，执行的sql和报错麻烦帮忙截图发下

#12

您好，创建用户时的报错截图为：

截屏2024-02-01 14.45.36
截屏2024-02-01 14.45.361920×273 54.3 KB

#13
  1. ob版本是多少？
    部署的方式是什么呢？
    建议还是改一下sysctl.conf文件
#14

ob版本是4.2.1.3；
部署方式使用的是 OBD 白屏部署
还有其他办法可以解决吗，之前好像还可以正常创建用户

#15

disk是否合标

#16

老师帮忙确认下：
1.具体OB的详细版本
2.执行语句对应的sql_audit记录文本
3.以及当时创建租户失败的时间段的OB日志

#17

老师，帮忙取下集群的基本信息：
一、需要您提供给我们集群相关基础信息(如部分已经提供则无需继续提供)：
1，root@sys租户登录集群，执行SQL提供下结果：
1），select svr_ip,zone,with_rootserver,status,block_migrate_in_time,start_service_time,stop_time,build_version from oceanbase.__all_server order by zone;
2），select tenant_id,tenant_name,primary_zone,compatibility_mode from oceanbase.__all_tenant;
3），show parameters like ‘%syslog_level%’;
4），show parameters like ‘%syslog_io_bandwidth_limit%’
5），select count(*),tenant_id,zone_list,unit_count from oceanbase.__all_resource_pool group by tenant_id,zone_list,unit_count;
2，登录OB任意一台主机，执行lsblk提供下结果
3，登录OB任意一台主机，执行lscpu | grep Architecture 提供下结果

然后我看报错是io报错，您帮忙取下observer日志和rs节点的rs日志看下（要取对应时间点的完整的日志）。

ob日志帮忙取下对应svr_ip节点的：

SET ob_enable_trace_log=‘ON’;
复现sql
SHOW TRACE; --得到trace_id

根据trace_id过滤下trace日志

老师再帮忙看下ocp上这个集群有什么告警吗

#18

好的好的！我尽快试一下保存日志

#19

前面几步操作结果的截图：
创建租户失败截图

失败截图
失败截图1920×614 116 KB

select svr_ip,zone, with_rootserver, status, block_migrate_in_time, start_service_time, stop_time, build_version from oceanbase.__all_server order by zone;截图：
res1
res11920×1170 232 KB

select tenant_id,tenant_name,primary_zone,compatibility_mode from oceanbase.__all_tenant; 截图
res2
res21920×493 75.9 KB

show parameters like ‘%syslog_level%’; 截图：
res3
res31920×1331 256 KB

show parameters like ‘%syslog_io_bandwidth_limit%’ 截图：
res4
res41920×1106 252 KB

select count(*),tenant_id,zone_list,unit_count from oceanbase.__all_resource_pool group by tenant_id,zone_list,unit_count; 截图：
res5
res51920×495 82.1 KB

lsblk结果：
res-lsblk
res-lsblk1240×1488 242 KB

lscpu结果：
res-lscpu
res-lscpu1258×214 93.2 KB

最后
set ob_enable_show_trace=1; 然后创建租户，show trace 时trace log没有看到输出

#20

image
image1102×302 9.97 KB

麻烦获取下日志。发下日志。

#21

为啥这样取不到log：
obclient [(none)]> set ob_enable_show_trace=1;
Query OK, 0 rows affected (0.000 sec)

obclient [(none)]> create tenant monitor resource_pool_list = (‘monitor_pool’), zone_list(‘zone1’, ‘zone2’, ‘zone3’), PRIMARY_ZONE=RANDOM set variables ob_compatibility_mode=‘mysql’, ob_tcp_invited_nodes=’%’ ;
ERROR 4009 (58030): IO error
obclient [(none)]> show trace ;
Empty set (0.011 sec)

obclient [(none)]>

#22

我从obd白屏上面创建租户的日志打出来：
############{EXECUTE}{2024-02-02T11:40:23.624+08:00}############2024-02-02 11:40:23.628 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.t.engine.runner.JavaSubtaskRunner : Run subtask, id=12846, context=Context{parallelIdx=-1, stringMap={tenant_name=monitor, ob_tenant_parameter_map=, prohibit_rollback=false, task_instance_id=12780, task_operation=execute, whitelist=%, target_tenant_status=NORMAL, old_password=, new_password=, tenant_mode=MYSQL, system_variable_map=, create_tenant_param_json={“charset”:“utf8mb4”,“collation”:“utf8mb4_general_ci”,“mode”:“MYSQL”,“name”:“monitor”,“parameters”:[],“primaryZone”:"",“rootPassword”:"",“whitelist”:"%",“zones”:[{“name”:“zone1”,“replicaType”:“FULL”,“resourcePool”:{“unitCount”:1,“unitSpec”:{“cpuCore”:3.00,“memoryBytes”:2147483648,“memorySize”:2}}},{“name”:“zone2”,“replicaType”:“FULL”,“resourcePool”:{“unitCount”:1,“unitSpec”:{“cpuCore”:3.00,“memoryBytes”:2147483648,“memorySize”:2}}},{“name”:“zone3”,“replicaType”:“FULL”,“resourcePool”:{“unitCount”:1,“unitSpec”:{“cpuCore”:3.00,“memoryBytes”:2147483648,“memorySize”:2}}}]}, latest_execution_start_time=2024-02-02T11:40:23.617+08:00, sub_task_instance_id=12846}, listMap={}}, executor=11.162.218.226
2024-02-02 11:40:23.631 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.o.i.tenant.task.CreateTenantTask : begin create tenant, param=CreateTenantParam(name=monitor, mode=MYSQL, primaryZone=, charset=utf8mb4, collation=utf8mb4_general_ci, description=null, whitelist=%, timeZone=null, rootPassword=, zones=[CreateTenantParam.ZoneParam(name=zone1, replicaType=FULL, resourcePool=CreateTenantParam.PoolParam(unitSpec=UnitSpecParam(cpuCore=3.0, memorySize=2), unitCount=1)), CreateTenantParam.ZoneParam(name=zone2, replicaType=FULL, resourcePool=CreateTenantParam.PoolParam(unitSpec=UnitSpecParam(cpuCore=3.0, memorySize=2), unitCount=1)), CreateTenantParam.ZoneParam(name=zone3, replicaType=FULL, resourcePool=CreateTenantParam.PoolParam(unitSpec=UnitSpecParam(cpuCore=3.0, memorySize=2), unitCount=1))], parameters=[])
2024-02-02 11:40:23.635 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.c.obsdk.ConnectPropertiesBuilder : build connect address for sys tenant, obServerAddrList=[ObServerAddr(address=11.162.218.35, port=2881), ObServerAddr(address=11.162.218.226, port=2881), ObServerAddr(address=11.162.218.58, port=2881)]
2024-02-02 11:40:23.639 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.642 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.644 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.648 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.651 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:40:23.654 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = ‘min_observer_version’
2024-02-02 11:40:23.660 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.663 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.666 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:40:23.670 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: CREATE RESOURCE UNIT config_monitor_zone2_3C2G_ksv MAX_CPU = ?, MIN_CPU = ?, MEMORY_SIZE = ?, args: [3.0, 3.0, 2147483648]
2024-02-02 11:40:23.680 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.684 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.687 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:40:23.691 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT UNIT_CONFIG_ID, NAME, MAX_CPU, MIN_CPU, MEMORY_SIZE AS MAX_MEMORY, MEMORY_SIZE AS MIN_MEMORY, LOG_DISK_SIZE, MAX_IOPS, MIN_IOPS, IOPS_WEIGHT FROM oceanbase.DBA_OB_UNIT_CONFIGS WHERE NAME = ?, args: [config_monitor_zone2_3C2G_ksv]
2024-02-02 11:40:23.701 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.705 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.708 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:40:23.711 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: CREATE RESOURCE POOL pool_monitor_zone2_ksv UNIT = ?, UNIT_NUM = ?, ZONE_LIST=(‘zone2’), args: [config_monitor_zone2_3C2G_ksv, 1]
2024-02-02 11:40:23.721 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.725 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.728 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:40:23.732 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.MAX_IOPS, t2.MIN_IOPS FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID WHERE t1.name = ?, args: [pool_monitor_zone2_ksv]
2024-02-02 11:40:23.747 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.o.i.tenant.task.CreateTenantTask : create resource pool success, resourcePoolList=[ResourcePool(id=1035, name=pool_monitor_zone1_sai, unitCount=1, unitConfig=UnitConfig(maxCpuCoreCount=3.0, minCpuCoreCount=3.0, maxMemoryByte=2147483648, minMemoryByte=2147483648, maxDiskSizeByte=null, maxDiskSize=null, maxIops=null, minIops=null, maxSessionNum=null, name=config_monitor_zone1_3C2G_sai), zoneList=[zone1]), ResourcePool(id=1034, name=pool_monitor_zone2_ksv, unitCount=1, unitConfig=UnitConfig(maxCpuCoreCount=3.0, minCpuCoreCount=3.0, maxMemoryByte=2147483648, minMemoryByte=2147483648, maxDiskSizeByte=null, maxDiskSize=null, maxIops=null, minIops=null, maxSessionNum=null, name=config_monitor_zone2_3C2G_ksv), zoneList=[zone2]), ResourcePool(id=1036, name=pool_monitor_zone3_rip, unitCount=1, unitConfig=UnitConfig(maxCpuCoreCount=3.0, minCpuCoreCount=3.0, maxMemoryByte=2147483648, minMemoryByte=2147483648, maxDiskSizeByte=null, maxDiskSize=null, maxIops=null, minIops=null, maxSessionNum=null, name=config_monitor_zone3_3C2G_rip), zoneList=[zone3])]
2024-02-02 11:40:23.751 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.c.obsdk.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test2024_lzh_ob, tenantName=sys, dbUser=root
2024-02-02 11:40:23.754 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.c.obsdk.ConnectPropertiesBuilder : build connect address for sys tenant, obServerAddrList=[ObServerAddr(address=11.162.218.35, port=2881), ObServerAddr(address=11.162.218.226, port=2881), ObServerAddr(address=11.162.218.58, port=2881)]
2024-02-02 11:40:23.757 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.760 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.763 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.765 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.768 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:40:23.771 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = ‘min_observer_version’
2024-02-02 11:40:23.777 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:40:23.781 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:40:23.783 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:40:23.786 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: CREATE TENANT monitor resource_pool_list=(‘pool_monitor_zone1_sai’,‘pool_monitor_zone2_ksv’,‘pool_monitor_zone3_rip’), LOCALITY = ?, CHARSET = ?, COLLATE = ? SET ob_tcp_invited_nodes=’%’, ob_compatibility_mode = ?, args: [FULL@zone1,FULL@zone2,FULL@zone3, utf8mb4, utf8mb4_general_ci, mysql]
2024-02-02 11:40:24.760 ERROR 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] update failed, sql:[CREATE TENANT monitor resource_pool_list=(‘pool_monitor_zone1_sai’,‘pool_monitor_zone2_ksv’,‘pool_monitor_zone3_rip’), LOCALITY = ?, CHARSET = ?, COLLATE = ? SET ob_tcp_invited_nodes=’%’, ob_compatibility_mode = ?], error message:[PreparedStatementCallback; SQL [CREATE TENANT monitor resource_pool_list=(‘pool_monitor_zone1_sai’,‘pool_monitor_zone2_ksv’,‘pool_monitor_zone3_rip’), LOCALITY = ?, CHARSET = ?, COLLATE = ? SET ob_tcp_invited_nodes=’%’, ob_compatibility_mode = ?]; (conn=3221699732) IO error; nested exception is java.sql.SQLTransientConnectionException: (conn=3221699732) IO error]
2024-02-02 11:40:24.765 WARN 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.o.task.engine.runner.RunnerFactory : Execute task failed, subtask=SubtaskInstanceOverview{id=12846, name=Create ob tenant, state=FAILED, operation=EXECUTE, className=com.oceanbase.ocp.obops.internal.tenant.task.CreateTenantTask, seriesId=1, startTime=2024-02-02T11:40:23.618+08:00, endTime=null}

com.oceanbase.ocp.obsdk.exception.OceanBaseException: (conn=3221699732) IO error
at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.customizeException(ConnectTemplate.java:342) ~[ocp-express-obsdk-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.customizeException(ConnectTemplate.java:334) ~[ocp-express-obsdk-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.customizeException(ConnectTemplate.java:329) ~[ocp-express-obsdk-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.updateInner(ConnectTemplate.java:207) ~[ocp-express-obsdk-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.update(ConnectTemplate.java:185) ~[ocp-express-obsdk-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.obsdk.operator.tenant.MysqlTenantOperator.createTenant(MysqlTenantOperator.java:119) ~[ocp-express-obsdk-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.obops.internal.tenant.TenantOperationServiceImpl.createTenantOnResourcePool(TenantOperationServiceImpl.java:169) ~[ocp-express-obops-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.obops.internal.tenant.TenantOperationServiceImpl.createTenantOnResourcePool(TenantOperationServiceImpl.java:148) ~[ocp-express-obops-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.obops.internal.tenant.task.CreateTenantTask.run(CreateTenantTask.java:72) ~[ocp-express-obops-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:63) ~[ocp-express-task-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) ~[ocp-express-task-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:29) ~[ocp-express-task-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:114) ~[ocp-express-task-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.task.engine.runner.RunnerFactory.redirectOutputIfNotSysSchedule(RunnerFactory.java:185) ~[ocp-express-task-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.task.engine.runner.RunnerFactory.run(RunnerFactory.java:104) ~[ocp-express-task-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at com.oceanbase.ocp.task.engine.coordinator.worker.subtask.ReadySubtaskWorker.lambda$null$1(ReadySubtaskWorker.java:112) ~[ocp-express-task-service-4.2.1-20231116.jar!/:4.2.1-20231116]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_252]
Caused by: java.sql.SQLTransientConnectionException: (conn=3221699732) IO error
at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.JDBC4PreparedStatement.executeUpdate(JDBC4PreparedStatement.java:195) ~[oceanbase-client-2.4.3.jar!/:na]
at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeUpdate(DruidPooledPreparedStatement.java:255) ~[druid-1.2.8.jar!/:1.2.8]
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) ~[spring-jdbc-5.3.27.jar!/:5.3.27]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-5.3.27.jar!/:5.3.27]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.27.jar!/:5.3.27]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) ~[spring-jdbc-5.3.27.jar!/:5.3.27]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1025) ~[spring-jdbc-5.3.27.jar!/:5.3.27]
at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.updateInner(ConnectTemplate.java:204) ~[ocp-express-obsdk-4.2.1-20231116.jar!/:4.2.1-20231116]
… 16 common frames omitted
Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: IO error
at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ~[oceanbase-client-2.4.3.jar!/:na]
… 25 common frames omitted
Caused by: java.sql.SQLException: IO error
at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) ~[oceanbase-client-2.4.3.jar!/:na]
at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ~[oceanbase-client-2.4.3.jar!/:na]
… 26 common frames omitted

Set state for subtask: 12846, operation:EXECUTE, state: FAILED############{ROLLBACK}{2024-02-02T11:44:53.167+08:00}############2024-02-02 11:44:53.170 WARN 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.t.engine.runner.JavaSubtaskRunner : Rollback subtask, id=12846, context=Context{parallelIdx=-1, stringMap={tenant_name=monitor, ob_tenant_parameter_map=, prohibit_rollback=false, task_instance_id=12780, task_operation=rollback, whitelist=%, target_tenant_status=NORMAL, resource_pool_list_json=[{“id”:1035,“name”:“pool_monitor_zone1_sai”,“unitConfig”:{“maxCpuCoreCount”:3.00,“maxMemoryByte”:2147483648,“maxMemorySize”:2,“minCpuCoreCount”:3.00,“minMemoryByte”:2147483648,“minMemorySize”:2},“unitCount”:1},{“id”:1034,“name”:“pool_monitor_zone2_ksv”,“unitConfig”:{“maxCpuCoreCount”:3.00,“maxMemoryByte”:2147483648,“maxMemorySize”:2,“minCpuCoreCount”:3.00,“minMemoryByte”:2147483648,“minMemorySize”:2},“unitCount”:1},{“id”:1036,“name”:“pool_monitor_zone3_rip”,“unitConfig”:{“maxCpuCoreCount”:3.00,“maxMemoryByte”:2147483648,“maxMemorySize”:2,“minCpuCoreCount”:3.00,“minMemoryByte”:2147483648,“minMemorySize”:2},“unitCount”:1}], old_password=, new_password=, tenant_mode=MYSQL, system_variable_map=, create_tenant_param_json={“charset”:“utf8mb4”,“collation”:“utf8mb4_general_ci”,“mode”:“MYSQL”,“name”:“monitor”,“parameters”:[],“primaryZone”:"",“rootPassword”:"******",“whitelist”:"%",“zones”:[{“name”:“zone1”,“replicaType”:“FULL”,“resourcePool”:{“unitCount”:1,“unitSpec”:{“cpuCore”:3.00,“memoryBytes”:2147483648,“memorySize”:2}}},{“name”:“zone2”,“replicaType”:“FULL”,“resourcePool”:{“unitCount”:1,“unitSpec”:{“cpuCore”:3.00,“memoryBytes”:2147483648,“memorySize”:2}}},{“name”:“zone3”,“replicaType”:“FULL”,“resourcePool”:{“unitCount”:1,“unitSpec”:{“cpuCore”:3.00,“memoryBytes”:2147483648,“memorySize”:2}}}]}, latest_execution_start_time=2024-02-02T11:44:53.161+08:00, sub_task_instance_id=12846}, listMap={}}, executor=11.162.218.226
2024-02-02 11:44:53.175 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.c.obsdk.ConnectPropertiesBuilder : build connect address for sys tenant, obServerAddrList=[ObServerAddr(address=11.162.218.35, port=2881), ObServerAddr(address=11.162.218.226, port=2881), ObServerAddr(address=11.162.218.58, port=2881)]
2024-02-02 11:44:53.178 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:44:53.181 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:44:53.184 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:44:53.187 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:44:53.191 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:44:53.195 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = ‘min_observer_version’
2024-02-02 11:44:53.201 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.o.o.r.MysqlResourceOperator : start to delete resource pool, pool_name = pool_monitor_zone2_ksv
2024-02-02 11:44:53.204 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=11.162.218.35, port=2881, database=oceanbase)
2024-02-02 11:44:53.207 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:11.162.218.226, sql port:2881
2024-02-02 11:44:53.210 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000]
2024-02-02 11:44:53.213 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: DROP RESOURCE POOL IF EXISTS pool_monitor_zone2_ksv
2024-02-02 11:44:53.221 WARN 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.o.i.t.ResourcePoolServiceImpl : delete resource unit config failed. unitConfigName=null, error message:input unit config name is empty
2024-02-02 11:44:53.224 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.o.i.t.ResourcePoolServiceImpl : resource pool deleted, poolName=pool_monitor_zone2_ksv, poolId=1034
2024-02-02 11:44:53.228 INFO 490524 — [pool-manual-subtask-executor16,da5e8e6eaea349b8,fee736e227d6] c.o.o.o.i.tenant.task.CreateTenantTask : rollback, delete resource pool, resourcePoolList=[ResourcePool(id=1035, name=pool_monitor_zone1_sai, unitCount=1, unitConfig=UnitConfig(maxCpuCoreCount=3.0, minCpuCoreCount=3.0, maxMemoryByte=2147483648, minMemoryByte=2147483648, maxDiskSizeByte=null, maxDiskSize=null, maxIops=null, minIops=null, maxSessionNum=null, name=null), zoneList=null), ResourcePool(id=1034, name=pool_monitor_zone2_ksv, unitCount=1, unitConfig=UnitConfig(maxCpuCoreCount=3.0, minCpuCoreCount=3.0, maxMemoryByte=2147483648, minMemoryByte=2147483648, maxDiskSizeByte=null, maxDiskSize=null, maxIops=null, minIops=null, maxSessionNum=null, name=null), zoneList=null), ResourcePool(id=1036, name=pool_monitor_zone3_rip, unitCount=1, unitConfig=UnitConfig(maxCpuCoreCount=3.0, minCpuCoreCount=3.0, maxMemoryByte=2147483648, minMemoryByte=2147483648, maxDiskSizeByte=null, maxDiskSize=null, maxIops=null, minIops=null, maxSessionNum=null, name=null), zoneList=null)]
Set state for subtask: 12846, operation:ROLLBACK, state: PENDING

#23

报错行

2024-02-02 11:40:24.760 ERROR 490524 --- [pool-manual-subtask-executor16,da5e8e6eaea349b8,33879543baa0] c.o.ocp.obsdk.connector.ConnectTemplate  : [obsdk] update failed, sql:[CREATE TENANT `monitor` resource_pool_list=('pool_monitor_zone1_sai','pool_monitor_zone2_ksv','pool_monitor_zone3_rip'), LOCALITY = ?, CHARSET = ?, COLLATE = ? SET ob_tcp_invited_nodes='%', ob_compatibility_mode = ?], error message:[PreparedStatementCallback; SQL [CREATE TENANT `monitor` resource_pool_list=('pool_monitor_zone1_sai','pool_monitor_zone2_ksv','pool_monitor_zone3_rip'), LOCALITY = ?, CHARSET = ?, COLLATE = ? SET ob_tcp_invited_nodes='%', ob_compatibility_mode = ?]; (conn=3221699732) IO error; nested exception is java.sql.SQLTransientConnectionException: (conn=3221699732) IO error]
#24

select last_trace_id();
定位具体的实现方式：
grep ‘trace id’ observer.log
试试

#25

这样啥东西也没有啊

#26

已解决，存在节点安装目录磁盘满导致。 清理部分日志后，创建正常。