执行任何命令都报ERROR 4012 (HY000): Timeout

AntTech_6SY3SD · 2025 年6 月 4 日 10:47

测试环境
OCP部署模式
版本：OceanBase_CE-v4.3.5.0

1、在OCP平台备份的时候报错：

2、使用root用户登录sys，设置ob_query_timeout报错：
mysql> SET GLOBAL ob_query_timeout = 10000000;
ERROR 4012 (HY000): Timeout

3、OCP平台集群状态不可用：

try_again · 2025 年6 月 4 日 10:54

先在observer服务器检查下服务进程、端口，observer.log是否正常打印。看起来是数据库服务异常了

AntTech_6SY3SD · 2025 年6 月 4 日 10:58

三个节点看着是正常的，端口都有。日志也一直有滚动

辞霜 · 2025 年6 月 4 日 10:59

用诊断工具obdiag analyze log 分析一下日志看https://www.oceanbase.com/docs/common-obdiag-cn-1000000002968720

AntTech_6SY3SD · 2025 年6 月 4 日 13:40

日志怎么给你，钉钉？

辞霜 · 2025 年6 月 4 日 13:53

附件发在帖子上就行

AntTech_6SY3SD · 2025 年6 月 4 日 13:59

Analyze OceanBase Online Log Summary:
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| Node | Status | FileName | First Found Time | ErrorCode | Message | Count |
+==============+===========+=================================================================================================+============================+=============+===================+=========+
| 172.16.6.172 | PASS | | | | | |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log | 2025-06-04 11:29:00.066568 | -4024 | Buffer not enough | 2584 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log | 2025-06-04 11:29:06.475633 | -4012 | Timeout | 6 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604105822894 | 2025-06-04 10:56:07.937618 | -4012 | Timeout | 8 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604110126043 | 2025-06-04 10:59:06.295850 | -4012 | Timeout | 9 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604110424847 | 2025-06-04 11:02:06.317462 | -4012 | Timeout | 9 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604110725648 | 2025-06-04 11:05:06.336240 | -4012 | Timeout | 9 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111023766 | 2025-06-04 11:08:06.346895 | -4012 | Timeout | 9 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111436589 | 2025-06-04 11:11:06.370756 | -4012 | Timeout | 12 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111436589 | 2025-06-04 11:11:40.418272 | -4024 | Buffer not enough | 455 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111928593 | 2025-06-04 11:14:36.683001 | -4024 | Buffer not enough | 2508 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111928593 | 2025-06-04 11:15:06.387758 | -4012 | Timeout | 15 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604112416570 | 2025-06-04 11:19:28.881818 | -4024 | Buffer not enough | 4590 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604112416570 | 2025-06-04 11:20:06.418311 | -4012 | Timeout | 15 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604112859836 | 2025-06-04 11:24:16.570687 | -4024 | Buffer not enough | 6606 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.173 | Completed | /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604112859836 | 2025-06-04 11:25:06.453969 | -4012 | Timeout | 12 |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+
| 172.16.6.174 | PASS | | | | | |
±-------------±----------±------------------------------------------------------------------------------------------------±---------------------------±------------±------------------±--------+

Details:

Node: 172.16.6.172
Status: PASS
FileName: None
First Found Time: None
ErrorCode: None
Message: None
Count: None
Last Found Time: None
Cause: None
Solution: None
Trace_IDS: None

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log
First Found Time: 2025-06-04 11:29:00.066568
ErrorCode: -4024
Message: Buffer not enough
Count: 2584
Last Found Time: 2025-06-04 11:30:31.835073
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log
First Found Time: 2025-06-04 11:29:06.475633
ErrorCode: -4012
Message: Timeout
Count: 6
Last Found Time: 2025-06-04 11:30:12.271992
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604105822894
First Found Time: 2025-06-04 10:56:07.937618
ErrorCode: -4012
Message: Timeout
Count: 8
Last Found Time: 2025-06-04 10:58:12.056651
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604110126043
First Found Time: 2025-06-04 10:59:06.295850
ErrorCode: -4012
Message: Timeout
Count: 9
Last Found Time: 2025-06-04 11:01:12.080615
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604110424847
First Found Time: 2025-06-04 11:02:06.317462
ErrorCode: -4012
Message: Timeout
Count: 9
Last Found Time: 2025-06-04 11:04:12.101129
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604110725648
First Found Time: 2025-06-04 11:05:06.336240
ErrorCode: -4012
Message: Timeout
Count: 9
Last Found Time: 2025-06-04 11:07:12.133452
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111023766
First Found Time: 2025-06-04 11:08:06.346895
ErrorCode: -4012
Message: Timeout
Count: 9
Last Found Time: 2025-06-04 11:10:12.156767
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111436589
First Found Time: 2025-06-04 11:11:06.370756
ErrorCode: -4012
Message: Timeout
Count: 12
Last Found Time: 2025-06-04 11:14:12.181773
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111436589
First Found Time: 2025-06-04 11:11:40.418272
ErrorCode: -4024
Message: Buffer not enough
Count: 455
Last Found Time: 2025-06-04 11:14:36.562160
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111928593
First Found Time: 2025-06-04 11:14:36.683001
ErrorCode: -4024
Message: Buffer not enough
Count: 2508
Last Found Time: 2025-06-04 11:19:28.492393
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604111928593
First Found Time: 2025-06-04 11:15:06.387758
ErrorCode: -4012
Message: Timeout
Count: 15
Last Found Time: 2025-06-04 11:19:12.211250
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604112416570
First Found Time: 2025-06-04 11:19:28.881818
ErrorCode: -4024
Message: Buffer not enough
Count: 4590
Last Found Time: 2025-06-04 11:24:16.379039
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604112416570
First Found Time: 2025-06-04 11:20:06.418311
ErrorCode: -4012
Message: Timeout
Count: 15
Last Found Time: 2025-06-04 11:24:12.231834
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604112859836
First Found Time: 2025-06-04 11:24:16.570687
ErrorCode: -4024
Message: Buffer not enough
Count: 6606
Last Found Time: 2025-06-04 11:28:59.104656
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.173
Status: Completed
FileName: /app/soft/obdiag_analyze_pack_20250604113026/remote_172_16_6_173/observer.log.20250604112859836
First Found Time: 2025-06-04 11:25:06.453969
ErrorCode: -4012
Message: Timeout
Count: 12
Last Found Time: 2025-06-04 11:28:12.260218
Cause: Internal Error
Solution: Contact OceanBase Support
Trace_IDS: [‘0-0000000000000000-0-0’]

Node: 172.16.6.174
Status: PASS
FileName: None
First Found Time: None
ErrorCode: None
Message: None
Count: None
Last Found Time: None
Cause: None
Solution: None
Trace_IDS: None

辞霜 · 2025 年6 月 4 日 16:28

能发附件么把obdiag巡检出来的结果发出来。再提供一份observer日志附件

AntTech_6SY3SD · 2025 年6 月 4 日 16:32

result_details.txt (13.1 KB)
observer.log.gz (9.3 MB)

辞霜 · 2025 年6 月 4 日 16:40

4024报错，buffer空间不足，查看这篇文章看看是否可以解决
https://www.oceanbase.com/knowledge-base/oceanbase-database-20000000055?back=kb

AntTech_6SY3SD · 2025 年6 月 4 日 16:46

现在集群操作不了命令。

是不是只能强制杀进程重启了

辞霜 · 2025 年6 月 4 日 17:22

ocp当前是否展示ob集群的磁盘使用情况，麻烦截图看一下。
再查一下节点的防火墙是否都关掉了

AntTech_6SY3SD · 2025 年6 月 4 日 17:32

ocp页面已经不显示集群资源信息了。

里面只有十来个G数据，应该不是磁盘问题。
防火墙是关着的

辞霜 · 2025 年6 月 4 日 17:38

生产环境么

辞霜 · 2025 年6 月 4 日 17:40

当前是否有10点的日志保留，帮忙拿一份10:04-10:30的日志

AntTech_6SY3SD · 2025 年6 月 4 日 17:53

测试环境

AntTech_6SY3SD · 2025 年6 月 4 日 17:57

没有observer的日志

辞霜 · 2025 年6 月 4 日 17:59

提供一份这个日志看看

AntTech_6SY3SD · 2025 年6 月 4 日 18:00

cat observer.log.wf.20250604102831178
[2025-06-04 10:26:11.649444] INFO New syslog file info: [address: “172.16.6.173:2882”, observer version: OceanBase_CE 4.3.5.0, revision: 100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5, sysname: Linux, os release: 5.15.0-125-generic, machine: x86_64, tz GMT offset: 08:00]

辞霜 · 2025 年6 月 4 日 18:06

抱歉这边需要非wf日志，有observer.log.202506041028这个日志保存么