租户连接时报错8001-Server is initializing

OceanBase 技术问题
, ,
#24

1、看日志报错是日志磁盘满了 可以查看一下参数配置
show parameters where name in (‘memory_limit’,‘memory_limit_percentage’,‘system_memory’,‘log_disk_size’,‘log_disk_percentage’,‘datafile_size’,‘datafile_disk_percentage’);
2、三个节点的observer.log日志发一下

1 个赞
#25

重启之后看到ocp的告警，153这台机器分别告警了1001租户合并失败，租户内存不足，租户日志空间不足，另外obdiag巡检结果有些需要特别重视的critical项，请先修正

1.1001 clog目录满
麻烦使用obdiag进行下根因分析
obdiag rca --scene=clog_disk_full

2.这个clog 目录 “/data/oceanbase/data/log1/obd/clog” 应该人为创建了一些其它文件，这是不允许的，麻烦执行下以下命令查询下

find #{log_dir_path} -type f -name ‘[^0-9]’ ! -name ‘.tmp’ ! -name '.flashback’ ! -name ‘meta’
#{log_dir_path}换成clog盘路径

3.磁盘iops较差，测试环境可先忽略

4.目录的文件系统类型不是所要求的xfs或者ext4，麻烦更改为要求类型，/data/oceanbase/data/1/obd/clog ， /data/oceanbase/data/log1/obd/clog

------------------------------------------------------------------+
| network.TCP-retransmission | [critical] [local] tsar is not installed. we can not check tcp retransmission.                                                                                                                                      |
|    clog.clog_disk_full     | [critical] [cluster:obcluster] The following tenants have experienced clog disk full: 1001. Please check by obdiag rca --scene=clog_disk_full                                                                       |
|  disk.clog_abnormal_file   | [critical] [local] Users are not allowed to modify or create in the clog folder, It will be causing observer startup failure. Files need be checked: /bin/bash:行1: /data/oceanbase/data/log1/obd/clog：是一个目录  |
|       disk.disk_iops       | [critical] [cluster:obcluster] These observer 16K IOPS are below 1024, please migrate as soon as possible. 172.18.90.153:2882, 172.18.90.154:2882, 172.18.90.152:2882                                               |
|    system.core_pattern     | [critical] [local] kernel.core_pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h , is not recommended for configuring functions other than the specified core path                                   |
| cluster.data_path_settings | [critical] [local] ip:172.18.90.152 ,log_dir_path: /data/oceanbase/data/1/obd/clog                                                                                                                                  |
|                            | /data/oceanbase/data/log1/obd/clog file_system is not xfs or ext4.                                                                                                                                                  |
|   cluster.task_opt_stat    | [critical] [cluster:obcluster] failed_scheduler_tenant_id is exists. Please check the tenant_ids: 1,1001,1002                                                                                                       |
+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

另外发下重启之后3个节点的observer.log

2 个赞
#26

学习了 :100:

#27

然后呢？

#28

赞一个！

#29

实践出真知，感谢分享实战经验

#30

赞一个！

#31

干货满满，受益匪浅

#32

内容很好

#33

宝贵的经验分享，谢谢！

#34

期待更多分享

#35

期待更多分享

#36

感谢作者的整理和分享