############{1}{2023-08-24T10:36:29+08:00}############ [2023-08-24 10:36:29,117] INFO - Dependencies all met for [2023-08-24 10:36:29,126] INFO - Dependencies all met for [2023-08-24 10:36:29,126] INFO - -------------------------------------------------------------------------------- [2023-08-24 10:36:29,126] INFO - Starting attempt 1 of 1 [2023-08-24 10:36:29,126] INFO - -------------------------------------------------------------------------------- [2023-08-24 10:36:29,139] INFO - Executing on 2023-08-24 02:36:25.769944+00:00 [2023-08-24 10:36:29,145] INFO - Started process 10795 to run task [2023-08-24 10:36:29,153] INFO - Running: ['airflow', 'tasks', 'run', 'init_server_with_tag', 'config_chrony', 'manual__2023-08-24T02:36:25.769944+00:00', '--job-id', '242', '--raw', '--subdir', 'DAGS_FOLDER/init_server_with_tag.py', '--cfg-path', '/tmp/tmpxzehccsk', '--error-file', '/tmp/tmpqjq9t940'] [2023-08-24 10:36:29,155] INFO - Job 242: Subtask config_chrony [2023-08-24 10:36:29,216] INFO - Running on host 39192ddffb99 [2023-08-24 10:36:29,278] WARNING - /usr/local/lib/python3.9/site-packages/airflow/models/renderedtifields.py:249 SAWarning: Coercing Subquery object into a select() for use in IN(); please pass a select() construct explicitly [2023-08-24 10:36:29,286] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_server_with_tag AIRFLOW_CTX_TASK_ID=config_chrony AIRFLOW_CTX_EXECUTION_DATE=2023-08-24T02:36:25.769944+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2023-08-24T02:36:25.769944+00:00 [2023-08-24 10:36:29,295] INFO - Using connection ID 'oat' for task execution. [2023-08-24 10:36:29,301] INFO - execute command on 192.168.61.213: chrony_server_list=(192.168.61.211) min_poll=4 max_poll=6 max_slew=500 as_server=False allow_list=() deny_list=() if ! which chronyd; then echo "chrony not exists, try to install it" for i in yum apt-get zypper; do which $i && install_cmd="$i" && break; done [ -z "$install_cmd" ] && { echo "No yum/apt-get/zypper command!"; exit 1; } $install_cmd install -y chrony || { echo "install chrony software failed!"; exit 1; } systemctl start chronyd fi systemctl is-enabled chronyd || systemctl enable chronyd cp -f /etc/chrony.conf /etc/chrony.conf.old if [ "$as_server" = "True" ]; then sed -i '/^[[:blank:]]*local/d ' /etc/chrony.conf sed -i "1ilocal stratum 10" /etc/chrony.conf sed -i '/^[[:blank:]]*deny/d ' /etc/chrony.conf for client in "${deny_list[@]}"; do sed -i "1ideny $client" /etc/chrony.conf done sed -i '/^[[:blank:]]*allow/d ' /etc/chrony.conf for client in "${allow_list[@]}"; do sed -i "1iallow $client" /etc/chrony.conf done fi sed -i '/^[[:blank:]]*maxslewrate/d ' /etc/chrony.conf sed -i "1imaxslewrate $max_slew " /etc/chrony.conf sed -i '/^[[:blank:]]*server/d ' /etc/chrony.conf for server in "${chrony_server_list[@]}"; do sed -i "1iserver $server minpoll $min_poll maxpoll $max_poll iburst" /etc/chrony.conf done sed -i '/^[[:blank:]]*makestep/d ' /etc/chrony.conf if which firewall-cmd && systemctl status --no-pager firewalld; then firewall-cmd --add-service=ntp --permanent firewall-cmd --reload fi systemctl restart chronyd && systemctl disable ntpd chronyc makestep for i in `seq 10`; do (chronyc -n sources | grep -E '\^\*' &>/dev/null) && echo 'set chrony server success!' && hwclock -w && exit sleep 1 done echo "chrony sync wait timeout (10s)!" exit 1 [2023-08-24 10:36:29,313] INFO - Connected (version 2.0, client OpenSSH_7.4) [2023-08-24 10:36:29,397] INFO - Authentication (password) successful! [2023-08-24 10:36:29,475] INFO - /usr/sbin/chronyd [2023-08-24 10:36:29,480] INFO - enabled [2023-08-24 10:36:29,495] INFO - /usr/bin/firewall-cmd [2023-08-24 10:36:29,501] INFO - ● firewalld.service - firewalld - dynamic firewall daemon [2023-08-24 10:36:29,501] INFO - Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) [2023-08-24 10:36:29,501] INFO - Active: inactive (dead) [2023-08-24 10:36:29,501] INFO - Docs: man:firewalld(1) [2023-08-24 10:36:29,676] INFO - 200 OK [2023-08-24 10:36:39,742] INFO - chrony sync wait timeout (10s)! [2023-08-24 10:36:39,744] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 179, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_server_with_tag.py", line 77, in config_chrony common.config_chrony_settings(ctx, logger) File "/oat/task_engine/plugins/common.py", line 881, in config_chrony_settings raise RuntimeError() RuntimeError [2023-08-24 10:36:39,752] INFO - Marking task as FAILED. dag_id=init_server_with_tag, task_id=config_chrony, execution_date=20230824T023625, start_date=20230824T023629, end_date=20230824T023639 [2023-08-24 10:36:39,816] ERROR - Failed to execute job 242 for task config_chrony (; 10795) [2023-08-24 10:36:39,836] INFO - Task exited with return code 1 [2023-08-24 10:36:39,872] INFO - 0 downstream tasks scheduled from follow-on schedule check ############{2}{2023-08-24T10:55:48+08:00}############ [2023-08-24 10:55:48,814] INFO - Dependencies all met for [2023-08-24 10:55:48,822] INFO - Dependencies all met for [2023-08-24 10:55:48,822] INFO - -------------------------------------------------------------------------------- [2023-08-24 10:55:48,823] INFO - Starting attempt 2 of 2 [2023-08-24 10:55:48,823] INFO - -------------------------------------------------------------------------------- [2023-08-24 10:55:48,833] INFO - Executing on 2023-08-24 02:36:25.769944+00:00 [2023-08-24 10:55:48,838] INFO - Started process 13206 to run task [2023-08-24 10:55:48,847] INFO - Running: ['airflow', 'tasks', 'run', 'init_server_with_tag', 'config_chrony', 'manual__2023-08-24T02:36:25.769944+00:00', '--job-id', '248', '--raw', '--subdir', 'DAGS_FOLDER/init_server_with_tag.py', '--cfg-path', '/tmp/tmp9wyxo6jf', '--error-file', '/tmp/tmp47a575ss'] [2023-08-24 10:55:48,851] INFO - Job 248: Subtask config_chrony [2023-08-24 10:55:48,932] INFO - Running on host 39192ddffb99 [2023-08-24 10:55:48,999] WARNING - /usr/local/lib/python3.9/site-packages/airflow/models/renderedtifields.py:249 SAWarning: Coercing Subquery object into a select() for use in IN(); please pass a select() construct explicitly [2023-08-24 10:55:49,012] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_server_with_tag AIRFLOW_CTX_TASK_ID=config_chrony AIRFLOW_CTX_EXECUTION_DATE=2023-08-24T02:36:25.769944+00:00 AIRFLOW_CTX_TRY_NUMBER=2 AIRFLOW_CTX_DAG_RUN_ID=manual__2023-08-24T02:36:25.769944+00:00 [2023-08-24 10:55:49,020] INFO - Using connection ID 'oat' for task execution. [2023-08-24 10:55:49,027] INFO - execute command on 192.168.61.213: chrony_server_list=(192.168.61.211) min_poll=4 max_poll=6 max_slew=500 as_server=False allow_list=() deny_list=() if ! which chronyd; then echo "chrony not exists, try to install it" for i in yum apt-get zypper; do which $i && install_cmd="$i" && break; done [ -z "$install_cmd" ] && { echo "No yum/apt-get/zypper command!"; exit 1; } $install_cmd install -y chrony || { echo "install chrony software failed!"; exit 1; } systemctl start chronyd fi systemctl is-enabled chronyd || systemctl enable chronyd cp -f /etc/chrony.conf /etc/chrony.conf.old if [ "$as_server" = "True" ]; then sed -i '/^[[:blank:]]*local/d ' /etc/chrony.conf sed -i "1ilocal stratum 10" /etc/chrony.conf sed -i '/^[[:blank:]]*deny/d ' /etc/chrony.conf for client in "${deny_list[@]}"; do sed -i "1ideny $client" /etc/chrony.conf done sed -i '/^[[:blank:]]*allow/d ' /etc/chrony.conf for client in "${allow_list[@]}"; do sed -i "1iallow $client" /etc/chrony.conf done fi sed -i '/^[[:blank:]]*maxslewrate/d ' /etc/chrony.conf sed -i "1imaxslewrate $max_slew " /etc/chrony.conf sed -i '/^[[:blank:]]*server/d ' /etc/chrony.conf for server in "${chrony_server_list[@]}"; do sed -i "1iserver $server minpoll $min_poll maxpoll $max_poll iburst" /etc/chrony.conf done sed -i '/^[[:blank:]]*makestep/d ' /etc/chrony.conf if which firewall-cmd && systemctl status --no-pager firewalld; then firewall-cmd --add-service=ntp --permanent firewall-cmd --reload fi systemctl restart chronyd && systemctl disable ntpd chronyc makestep for i in `seq 10`; do (chronyc -n sources | grep -E '\^\*' &>/dev/null) && echo 'set chrony server success!' && hwclock -w && exit sleep 1 done echo "chrony sync wait timeout (10s)!" exit 1 [2023-08-24 10:55:49,044] INFO - Connected (version 2.0, client OpenSSH_7.4) [2023-08-24 10:55:49,150] INFO - Authentication (password) successful! [2023-08-24 10:55:49,235] INFO - /usr/sbin/chronyd [2023-08-24 10:55:49,241] INFO - enabled [2023-08-24 10:55:49,256] INFO - /usr/bin/firewall-cmd [2023-08-24 10:55:49,265] INFO - ● firewalld.service - firewalld - dynamic firewall daemon [2023-08-24 10:55:49,265] INFO - Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) [2023-08-24 10:55:49,265] INFO - Active: inactive (dead) [2023-08-24 10:55:49,265] INFO - Docs: man:firewalld(1) [2023-08-24 10:55:49,429] INFO - 200 OK [2023-08-24 10:55:59,494] INFO - chrony sync wait timeout (10s)! [2023-08-24 10:55:59,496] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 179, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_server_with_tag.py", line 77, in config_chrony common.config_chrony_settings(ctx, logger) File "/oat/task_engine/plugins/common.py", line 881, in config_chrony_settings raise RuntimeError() RuntimeError [2023-08-24 10:55:59,508] INFO - Marking task as FAILED. dag_id=init_server_with_tag, task_id=config_chrony, execution_date=20230824T023625, start_date=20230824T025548, end_date=20230824T025559 [2023-08-24 10:55:59,520] ERROR - Failed to execute job 248 for task config_chrony (; 13206) [2023-08-24 10:55:59,556] INFO - Task exited with return code 1 [2023-08-24 10:55:59,592] INFO - 0 downstream tasks scheduled from follow-on schedule check