【 使用环境 】 测试环境
【 OB or 其他组件 】
【 使用版本 】
【问题描述】清晰明确描述问题
【复现路径】 [2023-12-12T13:47:38.054+0800] INFO - sysctl /proc/sys/fs/aio-max-nr = 1048576, correct … PASS
67
[2023-12-12T13:47:38.064+0800] INFO - sysctl /proc/sys/vm/overcommit_memory = 0, correct … PASS
68
[2023-12-12T13:47:38.076+0800] INFO - sysctl /proc/sys/vm/nr_hugepages = 0, correct … PASS
69
[2023-12-12T13:47:38.089+0800] INFO - sysctl /proc/sys/net/ipv4/ip_forward = 1, correct … PASS
70
[2023-12-12T13:47:38.101+0800] INFO - sysctl /proc/sys/net/ipv4/ip_local_port_range = 10000 65535, correct … PASS
71
[2023-12-12T13:47:38.193+0800] INFO - check service [crond]: enabled … PASS
72
[2023-12-12T13:47:38.361+0800] INFO - check service [sshd]: enabled … PASS
73
[2023-12-12T13:47:38.438+0800] INFO - check service [firewalld]: inactive … PASS
74
[2023-12-12T13:47:38.445+0800] INFO - check service [firewalld]: disabled … PASS
75
[2023-12-12T13:47:38.450+0800] INFO - check sshd_config PubkeyAuthentication: yes … PASS
76
[2023-12-12T13:47:38.454+0800] INFO - check sshd_config UseDNS: no … PASS
77
[2023-12-12T13:47:38.459+0800] INFO - check sshd_config ClientAliveInterval: 60 … PASS
78
[2023-12-12T13:47:38.464+0800] INFO - check sshd_config ClientAliveCountMax: 10 … PASS
79
[2023-12-12T13:47:38.466+0800] INFO - check hugepage: disabled … PASS
80
[2023-12-12T13:47:38.467+0800] INFO - check oceanbase_limits.conf, exist … PASS
81
[2023-12-12T13:47:38.571+0800] INFO - check hard limit of new session open_files (ulimit -H -n): 655360 … PASS
82
[2023-12-12T13:47:38.572+0800] INFO - check hard limit of open_files (ulimit -H -n): 655360 … PASS
83
[2023-12-12T13:47:38.625+0800] INFO - check soft limit of new session open_files (ulimit -S -n): 655360 … PASS
84
[2023-12-12T13:47:38.626+0800] INFO - check soft limit of open_files (ulimit -S -n): 655360 … PASS
85
[2023-12-12T13:47:38.690+0800] INFO - check hard limit of new session max_user_processes (ulimit -H -u): 655360 … PASS
86
[2023-12-12T13:47:38.691+0800] INFO - check hard limit of max_user_processes (ulimit -H -u): 655360 … PASS
87
[2023-12-12T13:47:38.738+0800] INFO - check soft limit of new session max_user_processes (ulimit -S -u): 655360 … PASS
88
[2023-12-12T13:47:38.739+0800] INFO - check soft limit of max_user_processes (ulimit -S -u): 655360 … PASS
89
[2023-12-12T13:47:38.794+0800] INFO - check hard limit of new session stack_size (ulimit -H -s): 10240 … PASS
90
[2023-12-12T13:47:38.794+0800] INFO - check hard limit of stack_size (ulimit -H -s): 10240 … PASS
91
[2023-12-12T13:47:38.842+0800] INFO - check soft limit of new session stack_size (ulimit -S -s): 10240 … PASS
92
[2023-12-12T13:47:38.843+0800] INFO - check soft limit of stack_size (ulimit -S -s): 10240 … PASS
93
[2023-12-12T13:47:38.911+0800] INFO - check hard limit of new session core_file_size (ulimit -H -c): unlimited … PASS
94
[2023-12-12T13:47:38.912+0800] INFO - check hard limit of core_file_size (ulimit -H -c): unlimited … PASS
95
[2023-12-12T13:47:38.965+0800] INFO - check soft limit of new session core_file_size (ulimit -S -c): unlimited … PASS
96
[2023-12-12T13:47:38.966+0800] INFO - check soft limit of core_file_size (ulimit -S -c): unlimited … PASS
97
[2023-12-12T13:47:39.030+0800] INFO - check hard limit of new session cpu_time (ulimit -H -t): unlimited … PASS
98
[2023-12-12T13:47:39.031+0800] INFO - check hard limit of cpu_time (ulimit -H -t): unlimited … PASS
99
[2023-12-12T13:47:39.086+0800] INFO - check soft limit of new session cpu_time (ulimit -S -t): unlimited … PASS
100
[2023-12-12T13:47:39.087+0800] INFO - check soft limit of cpu_time (ulimit -S -t): unlimited … PASS
101
[2023-12-12T13:47:39.096+0800] INFO - check numa stat, pass … PASS
102
[2023-12-12T13:47:39.114+0800] INFO - check elevator policy: deadline … PASS
103
[2023-12-12T13:47:39.116+0800] INFO - check current_clocksource: tsc … PASS
104
[2023-12-12T13:47:39.136+0800] INFO - check logical sector size of /dev/sda: 512 … PASS
105
[2023-12-12T13:47:39.137+0800] INFO - check logical sector size of /dev/sdb: 512 … PASS
106
[2023-12-12T13:47:40.003+0800] INFO - check RPM: mariadb-5.5.68-1.el7.x86_64 mariadb-libs-5.5.68-1.el7.x86_64 is installed … PASS
107
[2023-12-12T13:47:40.611+0800] INFO - check RPM: python-devel-2.7.5-94.el7_9.x86_64 is installed … PASS
108
[2023-12-12T13:47:41.219+0800] INFO - check RPM: net-tools-2.0-0.25.20131004git.el7.x86_64 is installed … PASS
109
[2023-12-12T13:47:41.805+0800] INFO - check RPM: mtr-0.85-7.el7.x86_64 is installed … PASS
110
[2023-12-12T13:47:42.398+0800] INFO - check RPM: selinux-policy-targeted-3.13.1-268.el7.noarch tar-1.26-35.el7.x86_64 is installed … PASS
111
[2023-12-12T13:47:42.972+0800] INFO - check RPM: binutils-2.27-44.base.el7_9.1.x86_64 is installed … PASS
112
[2023-12-12T13:47:43.583+0800] INFO - check RPM: bind-utils-9.11.4-26.P2.el7_9.15.x86_64 is installed … PASS
113
[2023-12-12T13:47:44.316+0800] INFO - check RPM: libaio-0.3.109-13.el7.x86_64 is installed … PASS
114
[2023-12-12T13:47:44.931+0800] INFO - check RPM: libcurl-7.29.0-59.el7_9.1.x86_64 curl-7.29.0-59.el7_9.1.x86_64 python-pycurl-7.19.0-19.el7.x86_64 is installed … PASS
115
[2023-12-12T13:47:45.531+0800] INFO - check RPM: libatomic-4.8.5-44.el7.x86_64 is installed … PASS
116
[2023-12-12T13:47:46.106+0800] INFO - check RPM: ncurses-base-5.9-14.20130511.el7_4.noarch irqbalance-1.0.7-12.el7.x86_64 perl-Encode-2.51-7.el7.x86_64 nmap-ncat-6.40-19.el7.x86_64 ncurses-5.9-14.20130511.el7_4.x86_64 qrencode-libs-3.4.1-3.el7.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 vim-enhanced-7.4.629-8.el7_9.x86_64 is installed … PASS
117
[2023-12-12T13:47:46.732+0800] INFO - check RPM: iproute-4.11.0-30.el7.x86_64 is installed … PASS
118
[2023-12-12T13:47:46.821+0800] INFO - check mysql client, working … PASS
119
[2023-12-12T13:47:46.835+0800] INFO - checking irq affinity …
120
[2023-12-12T13:47:46.850+0800] INFO - checking ens160 …
121
[2023-12-12T13:47:46.851+0800] INFO - Cannot get device channel parameters
122
[2023-12-12T13:47:46.852+0800] INFO - : Operation not supported
123
[2023-12-12T13:47:46.856+0800] INFO - Cannot get device channel parameters
124
[2023-12-12T13:47:46.856+0800] INFO - : Operation not supported
125
[2023-12-12T13:47:46.858+0800] INFO - check irq channels, NIC: ens160, Channel Combined: … PASS
126
[2023-12-12T13:47:46.884+0800] INFO - check irq affinity, NIC: ens160, smp_affinity count: 5 … PASS
127
[2023-12-12T13:47:46.899+0800] INFO - check irqbalance status: unknown … PASS
128
[2023-12-12T13:47:46.899+0800] INFO - check irqbalance service: disabled … PASS
129
[2023-12-12T13:47:46.934+0800] INFO -
130
[2023-12-12T13:47:46.935+0800] INFO -
131
[2023-12-12T13:47:46.935+0800] INFO - ### SUMMARY OF ISSUES IN PRE-CHECK ###
132
[2023-12-12T13:47:46.936+0800] INFO - check CPU count: 16 < 32 … EXPECT >= 32 … FAIL
133
[2023-12-12T13:47:46.936+0800] INFO - TIPS: replace another machine with more CPU
134
[2023-12-12T13:47:46.937+0800] INFO - check total MEM: 31 GB < 128 GB … EXPECT >= 128 GB … FAIL
135
[2023-12-12T13:47:46.937+0800] INFO - TIPS: replace another machine with more MEM
136
[2023-12-12T13:47:46.937+0800] INFO - check /data/1, NOT mounted … EXPECT mounted as individual disk … FAIL
137
[2023-12-12T13:47:46.937+0800] INFO - TIPS: re-part disk to mount /data/1
138
[2023-12-12T13:47:46.938+0800] INFO - check /data/log1, NOT mounted … EXPECT mounted as individual disk … FAIL
139
[2023-12-12T13:47:46.938+0800] INFO - TIPS: re-part disk to mount /data/log1
140
[2023-12-12T13:47:46.939+0800] INFO - execute command on 192.168.2.234:
141
rm -f /tmp/precheck.shRr4RBLRq
142
[2023-12-12T13:47:47.002+0800] ERROR - Task failed with exception
143
Traceback (most recent call last):
144
File “/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py”, line 217, in execute
145
return_value = super().execute(context)
146
File “/usr/local/lib/python3.9/site-packages/airflow/operators/python.py”, line 175, in execute
147
return_value = self.execute_callable()
148
File “/usr/local/lib/python3.9/site-packages/airflow/operators/python.py”, line 192, in execute_callable
149
return self.python_callable(*self.op_args, **self.op_kwargs)
150
File “/oat/task_engine/dags/init_server_with_tag.py”, line 79, in precheck
151
common.server_precheck(ctx, logger=logger)
152
File “/oat/task_engine/plugins/common.py”, line 1542, in server_precheck
153
raise RuntimeError(‘server precheck failed, please see the summary info above for details’)
154
RuntimeError: server precheck failed, please see the summary info above for details
155
[2023-12-12T13:47:47.012+0800] INFO - Marking task as FAILED. dag_id=init_server_with_tag, task_id=precheck, execution_date=20231212T054716, start_date=20231212T054732, end_date=20231212T054747
156
[2023-12-12T13:47:47.013+0800] INFO - Running statement: update oat_audit set status=‘failed’, update_time=utc_timestamp(), failed_reason=%s where id=%s, parameters: [‘failed task instance is init_server_with_tag__precheck__20231212 and exception information is server precheck failed, please see the summary info above for details’, 31]
157
[2023-12-12T13:47:47.014+0800] INFO - Rows affected: 1
158
[2023-12-12T13:47:47.028+0800] ERROR - Failed to execute job 102 for task precheck (server precheck failed, please see the summary info above for details; 16904)
159
[2023-12-12T13:47:47.075+0800] INFO - Task exited with return code 1
160
【附件及日志】
