############{1}{2024-08-15T15:43:17+08:00}############ [2024-08-15T15:43:17.107+0800] INFO - Dependencies all met for dep_context=non-requeueable deps ti= [2024-08-15T15:43:17.123+0800] INFO - Dependencies all met for dep_context=requeueable deps ti= [2024-08-15T15:43:17.123+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T15:43:17.124+0800] INFO - Starting attempt 1 of 1 [2024-08-15T15:43:17.124+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T15:43:17.150+0800] INFO - Executing on 2024-08-15 07:43:10.533611+00:00 [2024-08-15T15:43:17.155+0800] INFO - Started process 18057 to run task [2024-08-15T15:43:17.158+0800] INFO - Running: ['airflow', 'tasks', 'run', 'init_server_with_tag', 'config_chrony', 'manual__2024-08-15T07:43:10.533611+00:00', '--job-id', '503', '--raw', '--subdir', 'DAGS_FOLDER/init_server_with_tag.py', '--cfg-path', '/tmp/tmpu9765vf6'] [2024-08-15T15:43:17.160+0800] INFO - Job 503: Subtask config_chrony [2024-08-15T15:43:17.247+0800] INFO - Running on host localhost.localdomain [2024-08-15T15:43:17.351+0800] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_server_with_tag AIRFLOW_CTX_TASK_ID=config_chrony AIRFLOW_CTX_EXECUTION_DATE=2024-08-15T07:43:10.533611+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2024-08-15T07:43:10.533611+00:00 [2024-08-15T15:43:17.353+0800] INFO - Running statement: select oat_server.id, oat_credential.id as credential_id, ip, ssh_port, username, password, auth_type, key_data, passphrase from oat_server, oat_credential where oat_server.credential_id=oat_credential.id and oat_server.id=%s, parameters: [1] [2024-08-15T15:43:17.355+0800] INFO - Rows affected: 1 [2024-08-15T15:43:17.357+0800] INFO - execute command on 10.252.252.51: chrony_server_list=(10.252.252.51) min_poll=4 max_poll=6 max_slew=500 as_server=True allow_list=(10.252.252.64 10.252.252.63 10.252.252.62 10.252.252.61 10.252.252.60 10.252.252.59 10.252.252.58 10.252.252.57 10.252.252.56 10.252.252.55 10.252.252.54 10.252.252.53 10.252.252.52 10.252.252.37) deny_list=() if ! which chronyd; then echo "chrony not exists, try to install it" for i in yum apt-get zypper; do which $i && install_cmd="$i" && break; done [ -z "$install_cmd" ] && { echo "No yum/apt-get/zypper command!"; exit 1; } $install_cmd install -y chrony || { echo "install chrony software failed!"; exit 1; } systemctl start chronyd fi systemctl is-enabled chronyd || systemctl enable chronyd cp -f /etc/chrony.conf /etc/chrony.conf.old if [ "$as_server" = "True" ]; then sed -i '/^[[:blank:]]*local/d ' /etc/chrony.conf sed -i "1ilocal stratum 10" /etc/chrony.conf sed -i '/^[[:blank:]]*deny/d ' /etc/chrony.conf for client in "${deny_list[@]}"; do sed -i "1ideny $client" /etc/chrony.conf done sed -i '/^[[:blank:]]*allow/d ' /etc/chrony.conf for client in "${allow_list[@]}"; do sed -i "1iallow $client" /etc/chrony.conf done fi sed -i '/^[[:blank:]]*maxslewrate/d ' /etc/chrony.conf sed -i "1imaxslewrate $max_slew " /etc/chrony.conf sed -i '/^[[:blank:]]*server/d ' /etc/chrony.conf for server in "${chrony_server_list[@]}"; do sed -i "1iserver $server minpoll $min_poll maxpoll $max_poll iburst" /etc/chrony.conf done sed -i '/^[[:blank:]]*makestep/d ' /etc/chrony.conf if which firewall-cmd && systemctl status --no-pager firewalld; then firewall-cmd --add-service=ntp --permanent firewall-cmd --reload fi systemctl restart chronyd && systemctl disable ntpd 2>/dev/null for i in `seq 10`; do (chronyc -n sources | grep -E '\^\*' &>/dev/null) && echo 'set chrony server success!' && chronyc makestep && hwclock -w && exit sleep 3 echo 'wait chrony source ready...' done echo "chrony sync wait timeout (10s)!" exit 1 [2024-08-15T15:43:17.510+0800] INFO - /usr/sbin/chronyd [2024-08-15T15:43:17.521+0800] INFO - enabled [2024-08-15T15:43:17.572+0800] INFO - /usr/bin/firewall-cmd [2024-08-15T15:43:17.590+0800] INFO - ● firewalld.service - firewalld - dynamic firewall daemon [2024-08-15T15:43:17.590+0800] INFO - Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) [2024-08-15T15:43:17.591+0800] INFO - Active: inactive (dead) [2024-08-15T15:43:17.591+0800] INFO - Docs: man:firewalld(1) [2024-08-15T15:43:17.591+0800] INFO - [2024-08-15T15:43:17.591+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T15:43:17.591+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T15:43:17.591+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T15:43:17.591+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T15:43:17.591+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T15:43:17.592+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T15:43:17.592+0800] INFO - 8月 15 14:37:26 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?). [2024-08-15T15:43:17.592+0800] INFO - 8月 15 15:08:39 localhost.localdomain systemd[1]: Stopping firewalld - dynamic firewall daemon... [2024-08-15T15:43:17.592+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: firewalld.service: Succeeded. [2024-08-15T15:43:17.592+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: Stopped firewalld - dynamic firewall daemon. [2024-08-15T15:43:21.011+0800] INFO - wait chrony source ready... [2024-08-15T15:43:24.015+0800] INFO - wait chrony source ready... [2024-08-15T15:43:27.019+0800] INFO - wait chrony source ready... [2024-08-15T15:43:30.023+0800] INFO - wait chrony source ready... [2024-08-15T15:43:33.027+0800] INFO - wait chrony source ready... [2024-08-15T15:43:36.031+0800] INFO - wait chrony source ready... [2024-08-15T15:43:39.035+0800] INFO - wait chrony source ready... [2024-08-15T15:43:42.039+0800] INFO - wait chrony source ready... [2024-08-15T15:43:45.042+0800] INFO - wait chrony source ready... [2024-08-15T15:43:48.047+0800] INFO - wait chrony source ready... [2024-08-15T15:43:48.047+0800] INFO - chrony sync wait timeout (10s)! [2024-08-15T15:43:48.049+0800] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 217, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_server_with_tag.py", line 78, in config_chrony common.config_chrony_settings(ctx, logger) File "/oat/task_engine/plugins/common.py", line 1576, in config_chrony_settings raise RuntimeError(f'config chrony settings failed') RuntimeError: config chrony settings failed [2024-08-15T15:43:48.061+0800] INFO - Marking task as FAILED. dag_id=init_server_with_tag, task_id=config_chrony, execution_date=20240815T074310, start_date=20240815T074317, end_date=20240815T074348 [2024-08-15T15:43:48.092+0800] ERROR - Failed to execute job 503 for task config_chrony (config chrony settings failed; 18057) [2024-08-15T15:43:48.125+0800] INFO - Task exited with return code 1 [2024-08-15T15:43:48.152+0800] INFO - 0 downstream tasks scheduled from follow-on schedule check ############{2}{2024-08-15T15:49:10+08:00}############ [2024-08-15T15:49:10.820+0800] INFO - Dependencies all met for dep_context=non-requeueable deps ti= [2024-08-15T15:49:10.843+0800] INFO - Dependencies all met for dep_context=requeueable deps ti= [2024-08-15T15:49:10.843+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T15:49:10.843+0800] INFO - Starting attempt 2 of 2 [2024-08-15T15:49:10.843+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T15:49:10.865+0800] INFO - Executing on 2024-08-15 07:43:10.533611+00:00 [2024-08-15T15:49:10.870+0800] INFO - Started process 20459 to run task [2024-08-15T15:49:10.873+0800] INFO - Running: ['airflow', 'tasks', 'run', 'init_server_with_tag', 'config_chrony', 'manual__2024-08-15T07:43:10.533611+00:00', '--job-id', '571', '--raw', '--subdir', 'DAGS_FOLDER/init_server_with_tag.py', '--cfg-path', '/tmp/tmpf3e6xa3k'] [2024-08-15T15:49:10.875+0800] INFO - Job 571: Subtask config_chrony [2024-08-15T15:49:10.960+0800] INFO - Running on host localhost.localdomain [2024-08-15T15:49:11.065+0800] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_server_with_tag AIRFLOW_CTX_TASK_ID=config_chrony AIRFLOW_CTX_EXECUTION_DATE=2024-08-15T07:43:10.533611+00:00 AIRFLOW_CTX_TRY_NUMBER=2 AIRFLOW_CTX_DAG_RUN_ID=manual__2024-08-15T07:43:10.533611+00:00 [2024-08-15T15:49:11.067+0800] INFO - Running statement: select oat_server.id, oat_credential.id as credential_id, ip, ssh_port, username, password, auth_type, key_data, passphrase from oat_server, oat_credential where oat_server.credential_id=oat_credential.id and oat_server.id=%s, parameters: [1] [2024-08-15T15:49:11.069+0800] INFO - Rows affected: 1 [2024-08-15T15:49:11.071+0800] INFO - execute command on 10.252.252.51: chrony_server_list=(10.252.252.51) min_poll=4 max_poll=6 max_slew=500 as_server=True allow_list=(10.252.252.64 10.252.252.63 10.252.252.62 10.252.252.61 10.252.252.60 10.252.252.59 10.252.252.58 10.252.252.57 10.252.252.56 10.252.252.55 10.252.252.54 10.252.252.53 10.252.252.52 10.252.252.37) deny_list=() if ! which chronyd; then echo "chrony not exists, try to install it" for i in yum apt-get zypper; do which $i && install_cmd="$i" && break; done [ -z "$install_cmd" ] && { echo "No yum/apt-get/zypper command!"; exit 1; } $install_cmd install -y chrony || { echo "install chrony software failed!"; exit 1; } systemctl start chronyd fi systemctl is-enabled chronyd || systemctl enable chronyd cp -f /etc/chrony.conf /etc/chrony.conf.old if [ "$as_server" = "True" ]; then sed -i '/^[[:blank:]]*local/d ' /etc/chrony.conf sed -i "1ilocal stratum 10" /etc/chrony.conf sed -i '/^[[:blank:]]*deny/d ' /etc/chrony.conf for client in "${deny_list[@]}"; do sed -i "1ideny $client" /etc/chrony.conf done sed -i '/^[[:blank:]]*allow/d ' /etc/chrony.conf for client in "${allow_list[@]}"; do sed -i "1iallow $client" /etc/chrony.conf done fi sed -i '/^[[:blank:]]*maxslewrate/d ' /etc/chrony.conf sed -i "1imaxslewrate $max_slew " /etc/chrony.conf sed -i '/^[[:blank:]]*server/d ' /etc/chrony.conf for server in "${chrony_server_list[@]}"; do sed -i "1iserver $server minpoll $min_poll maxpoll $max_poll iburst" /etc/chrony.conf done sed -i '/^[[:blank:]]*makestep/d ' /etc/chrony.conf if which firewall-cmd && systemctl status --no-pager firewalld; then firewall-cmd --add-service=ntp --permanent firewall-cmd --reload fi systemctl restart chronyd && systemctl disable ntpd 2>/dev/null for i in `seq 10`; do (chronyc -n sources | grep -E '\^\*' &>/dev/null) && echo 'set chrony server success!' && chronyc makestep && hwclock -w && exit sleep 3 echo 'wait chrony source ready...' done echo "chrony sync wait timeout (10s)!" exit 1 [2024-08-15T15:49:11.213+0800] INFO - /usr/sbin/chronyd [2024-08-15T15:49:11.225+0800] INFO - enabled [2024-08-15T15:49:11.276+0800] INFO - /usr/bin/firewall-cmd [2024-08-15T15:49:11.296+0800] INFO - ● firewalld.service - firewalld - dynamic firewall daemon [2024-08-15T15:49:11.297+0800] INFO - Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) [2024-08-15T15:49:11.297+0800] INFO - Active: inactive (dead) [2024-08-15T15:49:11.297+0800] INFO - Docs: man:firewalld(1) [2024-08-15T15:49:11.297+0800] INFO - [2024-08-15T15:49:11.297+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:11.297+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:11.298+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:11.298+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:11.298+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:11.298+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:11.298+0800] INFO - 8月 15 14:37:26 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?). [2024-08-15T15:49:11.298+0800] INFO - 8月 15 15:08:39 localhost.localdomain systemd[1]: Stopping firewalld - dynamic firewall daemon... [2024-08-15T15:49:11.298+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: firewalld.service: Succeeded. [2024-08-15T15:49:11.299+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: Stopped firewalld - dynamic firewall daemon. [2024-08-15T15:49:14.731+0800] INFO - wait chrony source ready... [2024-08-15T15:49:17.735+0800] INFO - wait chrony source ready... [2024-08-15T15:49:20.739+0800] INFO - wait chrony source ready... [2024-08-15T15:49:23.743+0800] INFO - wait chrony source ready... [2024-08-15T15:49:26.747+0800] INFO - wait chrony source ready... [2024-08-15T15:49:29.750+0800] INFO - wait chrony source ready... [2024-08-15T15:49:32.754+0800] INFO - wait chrony source ready... [2024-08-15T15:49:35.758+0800] INFO - wait chrony source ready... [2024-08-15T15:49:38.761+0800] INFO - wait chrony source ready... [2024-08-15T15:49:41.765+0800] INFO - wait chrony source ready... [2024-08-15T15:49:41.765+0800] INFO - chrony sync wait timeout (10s)! [2024-08-15T15:49:41.768+0800] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 217, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_server_with_tag.py", line 78, in config_chrony common.config_chrony_settings(ctx, logger) File "/oat/task_engine/plugins/common.py", line 1576, in config_chrony_settings raise RuntimeError(f'config chrony settings failed') RuntimeError: config chrony settings failed [2024-08-15T15:49:41.779+0800] INFO - Marking task as FAILED. dag_id=init_server_with_tag, task_id=config_chrony, execution_date=20240815T074310, start_date=20240815T074910, end_date=20240815T074941 [2024-08-15T15:49:41.809+0800] ERROR - Failed to execute job 571 for task config_chrony (config chrony settings failed; 20459) [2024-08-15T15:49:41.827+0800] INFO - Task exited with return code 1 [2024-08-15T15:49:41.855+0800] INFO - 0 downstream tasks scheduled from follow-on schedule check ############{3}{2024-08-15T15:49:48+08:00}############ [2024-08-15T15:49:48.699+0800] INFO - Dependencies all met for dep_context=non-requeueable deps ti= [2024-08-15T15:49:48.714+0800] INFO - Dependencies all met for dep_context=requeueable deps ti= [2024-08-15T15:49:48.714+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T15:49:48.714+0800] INFO - Starting attempt 3 of 3 [2024-08-15T15:49:48.714+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T15:49:48.735+0800] INFO - Executing on 2024-08-15 07:43:10.533611+00:00 [2024-08-15T15:49:48.739+0800] INFO - Started process 20711 to run task [2024-08-15T15:49:48.742+0800] INFO - Running: ['airflow', 'tasks', 'run', 'init_server_with_tag', 'config_chrony', 'manual__2024-08-15T07:43:10.533611+00:00', '--job-id', '586', '--raw', '--subdir', 'DAGS_FOLDER/init_server_with_tag.py', '--cfg-path', '/tmp/tmpm03o_62x'] [2024-08-15T15:49:48.744+0800] INFO - Job 586: Subtask config_chrony [2024-08-15T15:49:48.835+0800] INFO - Running on host localhost.localdomain [2024-08-15T15:49:48.941+0800] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_server_with_tag AIRFLOW_CTX_TASK_ID=config_chrony AIRFLOW_CTX_EXECUTION_DATE=2024-08-15T07:43:10.533611+00:00 AIRFLOW_CTX_TRY_NUMBER=3 AIRFLOW_CTX_DAG_RUN_ID=manual__2024-08-15T07:43:10.533611+00:00 [2024-08-15T15:49:48.944+0800] INFO - Running statement: select oat_server.id, oat_credential.id as credential_id, ip, ssh_port, username, password, auth_type, key_data, passphrase from oat_server, oat_credential where oat_server.credential_id=oat_credential.id and oat_server.id=%s, parameters: [1] [2024-08-15T15:49:48.945+0800] INFO - Rows affected: 1 [2024-08-15T15:49:48.947+0800] INFO - execute command on 10.252.252.51: chrony_server_list=(10.252.252.51) min_poll=4 max_poll=6 max_slew=500 as_server=True allow_list=(10.252.252.64 10.252.252.63 10.252.252.62 10.252.252.61 10.252.252.60 10.252.252.59 10.252.252.58 10.252.252.57 10.252.252.56 10.252.252.55 10.252.252.54 10.252.252.53 10.252.252.52 10.252.252.37) deny_list=() if ! which chronyd; then echo "chrony not exists, try to install it" for i in yum apt-get zypper; do which $i && install_cmd="$i" && break; done [ -z "$install_cmd" ] && { echo "No yum/apt-get/zypper command!"; exit 1; } $install_cmd install -y chrony || { echo "install chrony software failed!"; exit 1; } systemctl start chronyd fi systemctl is-enabled chronyd || systemctl enable chronyd cp -f /etc/chrony.conf /etc/chrony.conf.old if [ "$as_server" = "True" ]; then sed -i '/^[[:blank:]]*local/d ' /etc/chrony.conf sed -i "1ilocal stratum 10" /etc/chrony.conf sed -i '/^[[:blank:]]*deny/d ' /etc/chrony.conf for client in "${deny_list[@]}"; do sed -i "1ideny $client" /etc/chrony.conf done sed -i '/^[[:blank:]]*allow/d ' /etc/chrony.conf for client in "${allow_list[@]}"; do sed -i "1iallow $client" /etc/chrony.conf done fi sed -i '/^[[:blank:]]*maxslewrate/d ' /etc/chrony.conf sed -i "1imaxslewrate $max_slew " /etc/chrony.conf sed -i '/^[[:blank:]]*server/d ' /etc/chrony.conf for server in "${chrony_server_list[@]}"; do sed -i "1iserver $server minpoll $min_poll maxpoll $max_poll iburst" /etc/chrony.conf done sed -i '/^[[:blank:]]*makestep/d ' /etc/chrony.conf if which firewall-cmd && systemctl status --no-pager firewalld; then firewall-cmd --add-service=ntp --permanent firewall-cmd --reload fi systemctl restart chronyd && systemctl disable ntpd 2>/dev/null for i in `seq 10`; do (chronyc -n sources | grep -E '\^\*' &>/dev/null) && echo 'set chrony server success!' && chronyc makestep && hwclock -w && exit sleep 3 echo 'wait chrony source ready...' done echo "chrony sync wait timeout (10s)!" exit 1 [2024-08-15T15:49:49.088+0800] INFO - /usr/sbin/chronyd [2024-08-15T15:49:49.099+0800] INFO - enabled [2024-08-15T15:49:49.150+0800] INFO - /usr/bin/firewall-cmd [2024-08-15T15:49:49.166+0800] INFO - ● firewalld.service - firewalld - dynamic firewall daemon [2024-08-15T15:49:49.166+0800] INFO - Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) [2024-08-15T15:49:49.167+0800] INFO - Active: inactive (dead) [2024-08-15T15:49:49.167+0800] INFO - Docs: man:firewalld(1) [2024-08-15T15:49:49.167+0800] INFO - [2024-08-15T15:49:49.167+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:49.167+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:49.167+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:49.167+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:49.167+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:49.168+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T15:49:49.168+0800] INFO - 8月 15 14:37:26 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?). [2024-08-15T15:49:49.168+0800] INFO - 8月 15 15:08:39 localhost.localdomain systemd[1]: Stopping firewalld - dynamic firewall daemon... [2024-08-15T15:49:49.168+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: firewalld.service: Succeeded. [2024-08-15T15:49:49.168+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: Stopped firewalld - dynamic firewall daemon. [2024-08-15T15:49:52.574+0800] INFO - wait chrony source ready... [2024-08-15T15:49:55.578+0800] INFO - wait chrony source ready... [2024-08-15T15:49:58.582+0800] INFO - wait chrony source ready... [2024-08-15T15:50:01.586+0800] INFO - wait chrony source ready... [2024-08-15T15:50:04.589+0800] INFO - wait chrony source ready... [2024-08-15T15:50:07.593+0800] INFO - wait chrony source ready... [2024-08-15T15:50:10.597+0800] INFO - wait chrony source ready... [2024-08-15T15:50:13.601+0800] INFO - wait chrony source ready... [2024-08-15T15:50:16.606+0800] INFO - wait chrony source ready... [2024-08-15T15:50:19.610+0800] INFO - wait chrony source ready... [2024-08-15T15:50:19.610+0800] INFO - chrony sync wait timeout (10s)! [2024-08-15T15:50:19.612+0800] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 217, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_server_with_tag.py", line 78, in config_chrony common.config_chrony_settings(ctx, logger) File "/oat/task_engine/plugins/common.py", line 1576, in config_chrony_settings raise RuntimeError(f'config chrony settings failed') RuntimeError: config chrony settings failed [2024-08-15T15:50:19.622+0800] INFO - Marking task as FAILED. dag_id=init_server_with_tag, task_id=config_chrony, execution_date=20240815T074310, start_date=20240815T074948, end_date=20240815T075019 [2024-08-15T15:50:19.650+0800] ERROR - Failed to execute job 586 for task config_chrony (config chrony settings failed; 20711) [2024-08-15T15:50:19.696+0800] INFO - Task exited with return code 1 [2024-08-15T15:50:19.721+0800] INFO - 0 downstream tasks scheduled from follow-on schedule check ############{4}{2024-08-15T15:59:38+08:00}############ [2024-08-15T15:59:38.923+0800] INFO - Dependencies all met for dep_context=non-requeueable deps ti= [2024-08-15T15:59:38.939+0800] INFO - Dependencies all met for dep_context=requeueable deps ti= [2024-08-15T15:59:38.939+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T15:59:38.939+0800] INFO - Starting attempt 4 of 4 [2024-08-15T15:59:38.939+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T15:59:38.960+0800] INFO - Executing on 2024-08-15 07:43:10.533611+00:00 [2024-08-15T15:59:38.966+0800] INFO - Started process 25019 to run task [2024-08-15T15:59:38.970+0800] INFO - Running: ['airflow', 'tasks', 'run', 'init_server_with_tag', 'config_chrony', 'manual__2024-08-15T07:43:10.533611+00:00', '--job-id', '701', '--raw', '--subdir', 'DAGS_FOLDER/init_server_with_tag.py', '--cfg-path', '/tmp/tmpx9hd8ht5'] [2024-08-15T15:59:38.972+0800] INFO - Job 701: Subtask config_chrony [2024-08-15T15:59:39.060+0800] INFO - Running on host localhost.localdomain [2024-08-15T15:59:39.163+0800] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_server_with_tag AIRFLOW_CTX_TASK_ID=config_chrony AIRFLOW_CTX_EXECUTION_DATE=2024-08-15T07:43:10.533611+00:00 AIRFLOW_CTX_TRY_NUMBER=4 AIRFLOW_CTX_DAG_RUN_ID=manual__2024-08-15T07:43:10.533611+00:00 [2024-08-15T15:59:39.166+0800] INFO - Running statement: select oat_server.id, oat_credential.id as credential_id, ip, ssh_port, username, password, auth_type, key_data, passphrase from oat_server, oat_credential where oat_server.credential_id=oat_credential.id and oat_server.id=%s, parameters: [1] [2024-08-15T15:59:39.167+0800] INFO - Rows affected: 1 [2024-08-15T15:59:39.169+0800] INFO - execute command on 10.252.252.51: chrony_server_list=(10.252.252.51) min_poll=4 max_poll=6 max_slew=500 as_server=True allow_list=(10.252.252.64 10.252.252.63 10.252.252.62 10.252.252.61 10.252.252.60 10.252.252.59 10.252.252.58 10.252.252.57 10.252.252.56 10.252.252.55 10.252.252.54 10.252.252.53 10.252.252.52 10.252.252.37) deny_list=() if ! which chronyd; then echo "chrony not exists, try to install it" for i in yum apt-get zypper; do which $i && install_cmd="$i" && break; done [ -z "$install_cmd" ] && { echo "No yum/apt-get/zypper command!"; exit 1; } $install_cmd install -y chrony || { echo "install chrony software failed!"; exit 1; } systemctl start chronyd fi systemctl is-enabled chronyd || systemctl enable chronyd cp -f /etc/chrony.conf /etc/chrony.conf.old if [ "$as_server" = "True" ]; then sed -i '/^[[:blank:]]*local/d ' /etc/chrony.conf sed -i "1ilocal stratum 10" /etc/chrony.conf sed -i '/^[[:blank:]]*deny/d ' /etc/chrony.conf for client in "${deny_list[@]}"; do sed -i "1ideny $client" /etc/chrony.conf done sed -i '/^[[:blank:]]*allow/d ' /etc/chrony.conf for client in "${allow_list[@]}"; do sed -i "1iallow $client" /etc/chrony.conf done fi sed -i '/^[[:blank:]]*maxslewrate/d ' /etc/chrony.conf sed -i "1imaxslewrate $max_slew " /etc/chrony.conf sed -i '/^[[:blank:]]*server/d ' /etc/chrony.conf for server in "${chrony_server_list[@]}"; do sed -i "1iserver $server minpoll $min_poll maxpoll $max_poll iburst" /etc/chrony.conf done sed -i '/^[[:blank:]]*makestep/d ' /etc/chrony.conf if which firewall-cmd && systemctl status --no-pager firewalld; then firewall-cmd --add-service=ntp --permanent firewall-cmd --reload fi systemctl restart chronyd && systemctl disable ntpd 2>/dev/null for i in `seq 10`; do (chronyc -n sources | grep -E '\^\*' &>/dev/null) && echo 'set chrony server success!' && chronyc makestep && hwclock -w && exit sleep 3 echo 'wait chrony source ready...' done echo "chrony sync wait timeout (10s)!" exit 1 [2024-08-15T15:59:39.311+0800] INFO - /usr/sbin/chronyd [2024-08-15T15:59:39.325+0800] INFO - enabled [2024-08-15T15:59:39.377+0800] INFO - /usr/bin/firewall-cmd [2024-08-15T15:59:39.394+0800] INFO - ● firewalld.service - firewalld - dynamic firewall daemon [2024-08-15T15:59:39.394+0800] INFO - Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) [2024-08-15T15:59:39.394+0800] INFO - Active: inactive (dead) [2024-08-15T15:59:39.394+0800] INFO - Docs: man:firewalld(1) [2024-08-15T15:59:39.395+0800] INFO - [2024-08-15T15:59:39.395+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T15:59:39.395+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T15:59:39.395+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T15:59:39.395+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T15:59:39.395+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T15:59:39.395+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T15:59:39.396+0800] INFO - 8月 15 14:37:26 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?). [2024-08-15T15:59:39.396+0800] INFO - 8月 15 15:08:39 localhost.localdomain systemd[1]: Stopping firewalld - dynamic firewall daemon... [2024-08-15T15:59:39.396+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: firewalld.service: Succeeded. [2024-08-15T15:59:39.396+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: Stopped firewalld - dynamic firewall daemon. [2024-08-15T15:59:42.807+0800] INFO - wait chrony source ready... [2024-08-15T15:59:45.811+0800] INFO - wait chrony source ready... [2024-08-15T15:59:48.815+0800] INFO - wait chrony source ready... [2024-08-15T15:59:51.819+0800] INFO - wait chrony source ready... [2024-08-15T15:59:54.822+0800] INFO - wait chrony source ready... [2024-08-15T15:59:57.826+0800] INFO - wait chrony source ready... [2024-08-15T16:00:00.830+0800] INFO - wait chrony source ready... [2024-08-15T16:00:03.834+0800] INFO - wait chrony source ready... [2024-08-15T16:00:06.838+0800] INFO - wait chrony source ready... [2024-08-15T16:00:09.842+0800] INFO - wait chrony source ready... [2024-08-15T16:00:09.842+0800] INFO - chrony sync wait timeout (10s)! [2024-08-15T16:00:09.844+0800] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 217, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_server_with_tag.py", line 78, in config_chrony common.config_chrony_settings(ctx, logger) File "/oat/task_engine/plugins/common.py", line 1576, in config_chrony_settings raise RuntimeError(f'config chrony settings failed') RuntimeError: config chrony settings failed [2024-08-15T16:00:09.855+0800] INFO - Marking task as FAILED. dag_id=init_server_with_tag, task_id=config_chrony, execution_date=20240815T074310, start_date=20240815T075938, end_date=20240815T080009 [2024-08-15T16:00:09.884+0800] ERROR - Failed to execute job 701 for task config_chrony (config chrony settings failed; 25019) [2024-08-15T16:00:09.932+0800] INFO - Task exited with return code 1 [2024-08-15T16:00:09.969+0800] INFO - 0 downstream tasks scheduled from follow-on schedule check ############{5}{2024-08-15T16:02:52+08:00}############ [2024-08-15T16:02:52.184+0800] INFO - Dependencies all met for dep_context=non-requeueable deps ti= [2024-08-15T16:02:52.200+0800] INFO - Dependencies all met for dep_context=requeueable deps ti= [2024-08-15T16:02:52.200+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T16:02:52.200+0800] INFO - Starting attempt 5 of 5 [2024-08-15T16:02:52.200+0800] INFO - -------------------------------------------------------------------------------- [2024-08-15T16:02:52.221+0800] INFO - Executing on 2024-08-15 07:43:10.533611+00:00 [2024-08-15T16:02:52.227+0800] INFO - Started process 26356 to run task [2024-08-15T16:02:52.230+0800] INFO - Running: ['airflow', 'tasks', 'run', 'init_server_with_tag', 'config_chrony', 'manual__2024-08-15T07:43:10.533611+00:00', '--job-id', '747', '--raw', '--subdir', 'DAGS_FOLDER/init_server_with_tag.py', '--cfg-path', '/tmp/tmpg_fggjil'] [2024-08-15T16:02:52.232+0800] INFO - Job 747: Subtask config_chrony [2024-08-15T16:02:52.321+0800] INFO - Running on host localhost.localdomain [2024-08-15T16:02:52.422+0800] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_server_with_tag AIRFLOW_CTX_TASK_ID=config_chrony AIRFLOW_CTX_EXECUTION_DATE=2024-08-15T07:43:10.533611+00:00 AIRFLOW_CTX_TRY_NUMBER=5 AIRFLOW_CTX_DAG_RUN_ID=manual__2024-08-15T07:43:10.533611+00:00 [2024-08-15T16:02:52.424+0800] INFO - Running statement: select oat_server.id, oat_credential.id as credential_id, ip, ssh_port, username, password, auth_type, key_data, passphrase from oat_server, oat_credential where oat_server.credential_id=oat_credential.id and oat_server.id=%s, parameters: [1] [2024-08-15T16:02:52.426+0800] INFO - Rows affected: 1 [2024-08-15T16:02:52.427+0800] INFO - execute command on 10.252.252.51: chrony_server_list=(10.252.252.51) min_poll=4 max_poll=6 max_slew=500 as_server=True allow_list=(10.252.252.64 10.252.252.63 10.252.252.62 10.252.252.61 10.252.252.60 10.252.252.59 10.252.252.58 10.252.252.57 10.252.252.56 10.252.252.55 10.252.252.54 10.252.252.53 10.252.252.52 10.252.252.37) deny_list=() if ! which chronyd; then echo "chrony not exists, try to install it" for i in yum apt-get zypper; do which $i && install_cmd="$i" && break; done [ -z "$install_cmd" ] && { echo "No yum/apt-get/zypper command!"; exit 1; } $install_cmd install -y chrony || { echo "install chrony software failed!"; exit 1; } systemctl start chronyd fi systemctl is-enabled chronyd || systemctl enable chronyd cp -f /etc/chrony.conf /etc/chrony.conf.old if [ "$as_server" = "True" ]; then sed -i '/^[[:blank:]]*local/d ' /etc/chrony.conf sed -i "1ilocal stratum 10" /etc/chrony.conf sed -i '/^[[:blank:]]*deny/d ' /etc/chrony.conf for client in "${deny_list[@]}"; do sed -i "1ideny $client" /etc/chrony.conf done sed -i '/^[[:blank:]]*allow/d ' /etc/chrony.conf for client in "${allow_list[@]}"; do sed -i "1iallow $client" /etc/chrony.conf done fi sed -i '/^[[:blank:]]*maxslewrate/d ' /etc/chrony.conf sed -i "1imaxslewrate $max_slew " /etc/chrony.conf sed -i '/^[[:blank:]]*server/d ' /etc/chrony.conf for server in "${chrony_server_list[@]}"; do sed -i "1iserver $server minpoll $min_poll maxpoll $max_poll iburst" /etc/chrony.conf done sed -i '/^[[:blank:]]*makestep/d ' /etc/chrony.conf if which firewall-cmd && systemctl status --no-pager firewalld; then firewall-cmd --add-service=ntp --permanent firewall-cmd --reload fi systemctl restart chronyd && systemctl disable ntpd 2>/dev/null for i in `seq 10`; do (chronyc -n sources | grep -E '\^\*' &>/dev/null) && echo 'set chrony server success!' && chronyc makestep && hwclock -w && exit sleep 3 echo 'wait chrony source ready...' done echo "chrony sync wait timeout (10s)!" exit 1 [2024-08-15T16:02:52.559+0800] INFO - /usr/sbin/chronyd [2024-08-15T16:02:52.570+0800] INFO - enabled [2024-08-15T16:02:52.615+0800] INFO - /usr/bin/firewall-cmd [2024-08-15T16:02:52.632+0800] INFO - ● firewalld.service - firewalld - dynamic firewall daemon [2024-08-15T16:02:52.632+0800] INFO - Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) [2024-08-15T16:02:52.632+0800] INFO - Active: inactive (dead) [2024-08-15T16:02:52.632+0800] INFO - Docs: man:firewalld(1) [2024-08-15T16:02:52.632+0800] INFO - [2024-08-15T16:02:52.632+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T16:02:52.632+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name. [2024-08-15T16:02:52.633+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T16:02:52.633+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name. [2024-08-15T16:02:52.633+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T16:02:52.633+0800] INFO - 8月 15 14:37:25 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name. [2024-08-15T16:02:52.633+0800] INFO - 8月 15 14:37:26 localhost.localdomain firewalld[8801]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?). [2024-08-15T16:02:52.633+0800] INFO - 8月 15 15:08:39 localhost.localdomain systemd[1]: Stopping firewalld - dynamic firewall daemon... [2024-08-15T16:02:52.633+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: firewalld.service: Succeeded. [2024-08-15T16:02:52.634+0800] INFO - 8月 15 15:08:43 localhost.localdomain systemd[1]: Stopped firewalld - dynamic firewall daemon. [2024-08-15T16:02:56.030+0800] INFO - wait chrony source ready... [2024-08-15T16:02:59.034+0800] INFO - wait chrony source ready... [2024-08-15T16:03:02.037+0800] INFO - wait chrony source ready... [2024-08-15T16:03:05.041+0800] INFO - wait chrony source ready... [2024-08-15T16:03:08.045+0800] INFO - wait chrony source ready... [2024-08-15T16:03:11.049+0800] INFO - wait chrony source ready... [2024-08-15T16:03:14.053+0800] INFO - wait chrony source ready... [2024-08-15T16:03:17.057+0800] INFO - wait chrony source ready... [2024-08-15T16:03:20.061+0800] INFO - wait chrony source ready... [2024-08-15T16:03:23.064+0800] INFO - wait chrony source ready... [2024-08-15T16:03:23.065+0800] INFO - chrony sync wait timeout (10s)! [2024-08-15T16:03:23.067+0800] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 217, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_server_with_tag.py", line 78, in config_chrony common.config_chrony_settings(ctx, logger) File "/oat/task_engine/plugins/common.py", line 1576, in config_chrony_settings raise RuntimeError(f'config chrony settings failed') RuntimeError: config chrony settings failed [2024-08-15T16:03:23.079+0800] INFO - Marking task as FAILED. dag_id=init_server_with_tag, task_id=config_chrony, execution_date=20240815T074310, start_date=20240815T080252, end_date=20240815T080323 [2024-08-15T16:03:23.107+0800] ERROR - Failed to execute job 747 for task config_chrony (config chrony settings failed; 26356) [2024-08-15T16:03:23.151+0800] INFO - Task exited with return code 1 [2024-08-15T16:03:23.190+0800] INFO - 0 downstream tasks scheduled from follow-on schedule check