ob机器故障后加回集群但失败

OceanBase 技术问题
#1

【使用环境 】生产环境
【 OB or 其他组件 】OB
【 使用版本 】4.3.5
【问题描述】

image
image2188×1115 369 KB

81.69昨天掉盘了，ocp上执行的从集群删除节点成功，今天盘修好了打算加回集群，报错机器非空不让加，现在是需要清理哪里的东西吗？

5 个赞
#3

顶顶

#4

OceanBase 社区已接收您的帖子，正在跟进中。

#5

检查OB 运行用户对新数据目录拥有读写权限。

#6

点赞~~

#7

not empty server，机器不为空，是不是要先清理下

#8

系统干掉重新装，按全新的搞

#9

555

#10

顶顶顶

#11

看看目录/data/1与data/log1 下面有数据吗？如果有就清除！

#12

系统数据追平操作先

#13

感觉是之前的删除节点没清理干净

#14

1、原因是:192.168.81.69 上 observer 本地存储已有数据，因此内核拒绝加入。
Cannot create symbolic link from /data/1/archive/sstable
to /opt/oceanbase/oceanbase/store/archive/sstable
reason: symlink … file exists

结论:之前目录没有删除干净,删除干净以后添加没有问题
image

2、重新添加节点磁盘的ops没有过去

image
image2051×865 153 KB

[manual-subtask-executor-16,002167f2a2623e40,110e1dedb56d6df5] c.o.o.c.h.s.HostOperationServiceImpl : check io bench output failed. The current data disk io_ps is 964.0. Please ensure that the data disk io_ps >= 1000 and the log disk io_ps >= 2000
SELECT * FROM oceanbase.GV$OB_IO_CALIBRATION_STATUS;
±--------------±---------±-------------±--------------±---------------------------±---------------------------+
| SVR_IP | SVR_PORT | STORAGE_NAME | STATUS | START_TIME | FINISH_TIME |
±--------------±---------±-------------±--------------±---------------------------±---------------------------+
| 192.168.81.32 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.33 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.31 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.41 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.38 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.36 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.34 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.35 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.37 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.39 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.46 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.45 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.42 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.55 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.57 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.44 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.43 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.49 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.56 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.69 | 2882 | DATA | READY | 2026-06-11 18:19:10.109492 | 2026-06-11 18:20:52.404306 |
| 192.168.81.48 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.54 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.59 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
| 192.168.81.47 | 2882 | DATA | NOT AVAILABLE | NULL | NULL |
±--------------±---------±-------------±--------------±---------------------------±---------------------------+
SELECT * FROM oceanbase.GV$OB_IO_BENCHMARK where svr_ip=‘192.168.81.69’;
±--------------±---------±-------------±------±--------±-----±-----±--------+
| SVR_IP | SVR_PORT | STORAGE_NAME | MODE | SIZE | IOPS | MBPS | LATENCY |
±--------------±---------±-------------±------±--------±-----±-----±--------+
| 192.168.81.69 | 2882 | DATA | READ | 4096 | 964 | 3 | 16851 |
| 192.168.81.69 | 2882 | DATA | READ | 8192 | 823 | 6 | 19659 |
| 192.168.81.69 | 2882 | DATA | READ | 16384 | 738 | 11 | 21959 |
| 192.168.81.69 | 2882 | DATA | READ | 32768 | 1020 | 31 | 15852 |
| 192.168.81.69 | 2882 | DATA | READ | 65536 | 1404 | 87 | 11438 |
| 192.168.81.69 | 2882 | DATA | READ | 131072 | 877 | 109 | 18326 |
| 192.168.81.69 | 2882 | DATA | READ | 262144 | 473 | 118 | 34057 |
| 192.168.81.69 | 2882 | DATA | READ | 524288 | 308 | 154 | 52772 |
| 192.168.81.69 | 2882 | DATA | READ | 1048576 | 170 | 170 | 94956 |
| 192.168.81.69 | 2882 | DATA | READ | 2097152 | 155 | 310 | 104122 |
| 192.168.81.69 | 2882 | DATA | WRITE | 4096 | 1775 | 6 | 9020 |
| 192.168.81.69 | 2882 | DATA | WRITE | 8192 | 2755 | 21 | 5823 |
| 192.168.81.69 | 2882 | DATA | WRITE | 16384 | 626 | 9 | 25546 |
| 192.168.81.69 | 2882 | DATA | WRITE | 32768 | 1318 | 41 | 12157 |
| 192.168.81.69 | 2882 | DATA | WRITE | 65536 | 724 | 45 | 22365 |
| 192.168.81.69 | 2882 | DATA | WRITE | 131072 | 631 | 78 | 25383 |
| 192.168.81.69 | 2882 | DATA | WRITE | 262144 | 502 | 125 | 32185 |
| 192.168.81.69 | 2882 | DATA | WRITE | 524288 | 302 | 151 | 53093 |
| 192.168.81.69 | 2882 | DATA | WRITE | 1048576 | 207 | 207 | 77646 |
| 192.168.81.69 | 2882 | DATA | WRITE | 2097152 | 149 | 298 | 108524 |
±--------------±---------±-------------±------±--------±-----±-----±--------+
3、手动执行
ALTER SYSTEM RUN JOB “io_calibration” SERVER = ‘192.168.81.69:2882’;
4、检查iops信息
obclient(root@sys)[(none)]> SELECT * FROM oceanbase.GV$OB_IO_BENCHMARK where svr_ip=‘192.168.81.69’;
±--------------±---------±-------------±------±--------±-----±-----±--------+
| SVR_IP | SVR_PORT | STORAGE_NAME | MODE | SIZE | IOPS | MBPS | LATENCY |
±--------------±---------±-------------±------±--------±-----±-----±--------+
| 192.168.81.69 | 2882 | DATA | READ | 4096 | 5103 | 19 | 3136 |
| 192.168.81.69 | 2882 | DATA | READ | 8192 | 5960 | 46 | 2687 |
| 192.168.81.69 | 2882 | DATA | READ | 16384 | 5181 | 80 | 3090 |
| 192.168.81.69 | 2882 | DATA | READ | 32768 | 3984 | 124 | 4019 |
| 192.168.81.69 | 2882 | DATA | READ | 65536 | 2682 | 167 | 5974 |
| 192.168.81.69 | 2882 | DATA | READ | 131072 | 1650 | 206 | 9715 |
| 192.168.81.69 | 2882 | DATA | READ | 262144 | 926 | 231 | 17317 |
| 192.168.81.69 | 2882 | DATA | READ | 524288 | 608 | 304 | 26391 |
| 192.168.81.69 | 2882 | DATA | READ | 1048576 | 407 | 407 | 39413 |
| 192.168.81.69 | 2882 | DATA | READ | 2097152 | 294 | 588 | 54602 |
| 192.168.81.69 | 2882 | DATA | WRITE | 4096 | 7985 | 31 | 2004 |
| 192.168.81.69 | 2882 | DATA | WRITE | 8192 | 4409 | 34 | 3629 |
| 192.168.81.69 | 2882 | DATA | WRITE | 16384 | 3649 | 57 | 4387 |
| 192.168.81.69 | 2882 | DATA | WRITE | 32768 | 3055 | 95 | 5237 |
| 192.168.81.69 | 2882 | DATA | WRITE | 65536 | 2129 | 133 | 7523 |
| 192.168.81.69 | 2882 | DATA | WRITE | 131072 | 1287 | 160 | 12445 |
| 192.168.81.69 | 2882 | DATA | WRITE | 262144 | 750 | 187 | 21385 |
| 192.168.81.69 | 2882 | DATA | WRITE | 524288 | 431 | 215 | 37194 |
| 192.168.81.69 | 2882 | DATA | WRITE | 1048576 | 282 | 282 | 57039 |
| 192.168.81.69 | 2882 | DATA | WRITE | 2097152 | 207 | 414 | 77704 |
±--------------±---------±-------------±------±--------±-----±-----±--------+
20 rows in set (0.041 sec)

5、ocp重试以后过去了