clockdiff is down问题分析

OceanBase 技术问题
#1

clockdiff 命令执行不稳定，高频率返回down的结果。

环境：VMWARE 虚拟机 rocky9.5， 虚拟机之间网络ping测试正常。

故障情况如下：


[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
.............................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
............................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
..................................................
host=192.168.169.41 rtt=0(0)ms/0ms delta=0ms/0ms Wed Nov 19 11:53:16 2025[admin@rocky95 ~]$
[admin@rocky95 ~]$
[admin@rocky95 ~]$
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
.................................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
.............................................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
............................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
.............................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
............................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
............................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
............................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
....................................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
............................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
...........................clockdiff: 192.168.169.41 is down
[admin@rocky95 ~]$ clockdiff -o 192.168.169.41
..................................................
host=192.168.169.41 rtt=0(0)ms/0ms delta=0ms/0ms Wed Nov 19 11:54:31 2025[admin@rocky95 ~]$

相关检查如下：

[admin@rocky95 ~]$ command -v clockdiff
/usr/bin/clockdiff
[admin@rocky95 ~]$ which clockdiff
/usr/bin/clockdiff
[admin@rocky95 ~]$ clockdiff -V
clockdiff from iputils 20210202


[admin@rocky95 ~]$ clockdiff localhost
..................................................
host=localhost rtt=0(0)ms/0ms delta=0ms/0ms Wed Nov 19 11:59:23 2025[admin@rocky95 ~]$ clockdiff localhost
..................................................
host=localhost rtt=0(0)ms/0ms delta=0ms/0ms Wed Nov 19 11:59:26 2025[admin@rocky95 ~]$ clockdiff localhost
..................................................
host=localhost rtt=0(0)ms/0ms delta=0ms/0ms Wed Nov 19 11:59:27 2025[admin@rocky95 ~]$


[admin@rocky95 ~]$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 126973
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 655350
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 655360
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
[admin@rocky95 ~]$ getenforce
Disabled


[admin@rocky95 ~]$ sudo chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 192.168.169.41                5   9   377   375    -20us[  -93us] +/-   40ms
1 个赞
#3

clockdiff（Linux iputils 套件自带）用的是 ICMP Timestamp Request/Reply 探测远端时间并计算本地与它的差值，可选 UDP 模式。
出现 down，常见原因有：

  1. 网络不通
  • 目标 IP 无法 ping 通（主机关机、网段不通、路由配置错误）。
  1. 防火墙/安全策略阻止
  • 目标机或中间设备拦截了 ICMP（尤其是 Timestamp 类型），clockdiff 收不到回应就报 “down”。
  • 云服务安全组只允许 ping（Echo），但不放行 Timestamp 类型。
  1. 目标系统内核禁用了 ICMP Timestamp 回复
  • 很多现代 Linux 默认不响应 timestamp，因为该功能几乎不用且有安全隐患。
  1. 错误的 IP 或协议不匹配
  • 输入地址错误，或者目标只支持 IPv6 而你用 IPv4。
#4

找到一官方文档，但照做未发现问题；
clockdiff: xx.xx.xx.xx is down

#5
  1. 网络不通
  • 目标 IP 无法 ping 通（主机关机、网段不通、路由配置错误）。
    测试网络通 

[root@rocky95 ~]# chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 192.168.169.41                5   9   377   130   +784ns[+1149ns] +/-   39ms
[root@rocky95 ~]#  ping -T tsandaddr 192.168.169.41 -c 2
PING 192.168.169.41 (192.168.169.41) 56(124) bytes of data.
64 bytes from 192.168.169.41: icmp_seq=1 ttl=64 time=0.322 ms
TS:     192.168.169.53  28844981 absolute
        192.168.169.41  0
        192.168.169.41  0
        192.168.169.53  0

64 bytes from 192.168.169.41: icmp_seq=2 ttl=64 time=0.527 ms
TS:     192.168.169.53  28846042 absolute
        192.168.169.41  1
        192.168.169.41  0
        192.168.169.53  0


--- 192.168.169.41 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1061ms
rtt min/avg/max/mdev = 0.322/0.424/0.527/0.102 ms
  1. 防火墙/安全策略阻止
  • 目标机或中间设备拦截了 ICMP（尤其是 Timestamp 类型），clockdiff 收不到回应就报 “down”。
  • 云服务安全组只允许 ping（Echo），但不放行 Timestamp 类型。

防火墙已关闭

  1. 目标系统内核禁用了 ICMP Timestamp 回复
  • 很多现代 Linux 默认不响应 timestamp，因为该功能几乎不用且有安全隐患。
    大部分失败，有成功案例 

[root@rocky95 ~]# sysctl -a |grep icmp
net.ipv4.icmp_echo_enable_probe = 0
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_msgs_burst = 50
net.ipv4.icmp_msgs_per_sec = 1000
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168
net.ipv6.icmp.echo_ignore_all = 0
net.ipv6.icmp.echo_ignore_anycast = 0
net.ipv6.icmp.echo_ignore_multicast = 0
net.ipv6.icmp.ratelimit = 1000
net.ipv6.icmp.ratemask = 0-1,3-127
  1. 错误的 IP 或协议不匹配
  • 输入地址错误，或者目标只支持 IPv6 而你用 IPv4。
    IP正确，有成功案例
#6

确认下53节点时差是不是与其他节点很大

#7 
[root@rocky95 ~]# chronyc tracking
Reference ID    : C0A8A929 (192.168.169.41)
Stratum         : 6
Ref time (UTC)  : Thu Nov 20 07:58:15 2025
System time     : 0.000000000 seconds fast of NTP time
Last offset     : +0.000000365 seconds
RMS offset      : 0.000028641 seconds
Frequency       : 8.507 ppm fast
Residual freq   : -0.000 ppm
Skew            : 0.033 ppm
Root delay      : 0.070013240 seconds
Root dispersion : 0.003774045 seconds
Update interval : 517.3 seconds
Leap status     : Normal
[root@rocky95 ~]# chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 192.168.169.41                5   9   377   420   +784ns[+1149ns] +/-   39ms
#8

测试并抓包，请大佬帮忙看看：

[root@rocky95 ~]# tcpdump -i any  -vvv -w clockdi555.pcap
tcpdump: data link type LINUX_SLL2
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
^C287 packets captured
296 packets received by filter
0 packets dropped by kernel

[root@rocky95 ~]# clockdiff -o 192.168.169.41
............................clockdiff: 192.168.169.41 is down

可见最后十几个包超时

企业微信截图_17636305158711
企业微信截图_176363051587112234×902 187 KB

clockdi555.rar (16.4 KB)

#9

rocky95自带clockdiff版本较低，执行有BUG；


[root@rocky95 soft]# clockdiff -V
clockdiff from iputils 20210202

更新clockdiff新版本后解决问题

[root@rocky95 soft]# ./clockdiff -V
clockdiff from iputils 20240117
libcap: yes, IDN: yes, NLS: no, error.h: yes, getrandom(): yes, __fpending(): yes
[root@rocky95 soft]# ./clockdiff -o 192.168.169.41
..................................................
host=192.168.169.41 rtt=0(0)ms/0ms delta=0ms/0ms Thu Nov 20 17:32:09 2025
[root@rocky95 soft]# ./clockdiff -o 192.168.169.41
..................................................
host=192.168.169.41 rtt=0(0)ms/0ms delta=0ms/0ms Thu Nov 20 17:32:16 2025
[root@rocky95 soft]# ./clockdiff -o 192.168.169.41
..................................................
host=192.168.169.41 rtt=0(0)ms/0ms delta=0ms/0ms Thu Nov 20 17:32:20 2025
[root@rocky95 soft]# ./clockdiff -o 192.168.169.41
..................................................
host=192.168.169.41 rtt=0(0)ms/0ms delta=0ms/0ms Thu Nov 20 17:32:24 2025

BUG情况见：
Clockdiff host is down #326

#10

感谢反馈

#11

厉害