[2024-03-06T00:06:42.024+0800] INFO - Dependencies all met for [2024-03-06T00:06:42.050+0800] INFO - Dependencies all met for [2024-03-06T00:06:42.051+0800] INFO - -------------------------------------------------------------------------------- [2024-03-06T00:06:42.051+0800] INFO - Starting attempt 10 of 10 [2024-03-06T00:06:42.052+0800] INFO - -------------------------------------------------------------------------------- [2024-03-06T00:06:42.080+0800] INFO - Executing on 2024-02-07 07:01:06.501917+00:00 [2024-03-06T00:06:42.087+0800] INFO - Started process 9672 to run task [2024-03-06T00:06:42.096+0800] INFO - Running: ['airflow', 'tasks', 'run', 'init_ocp', 'init_ocp_meta', 'manual__2024-02-07T07:01:06.501917+00:00', '--job-id', '936', '--raw', '--subdir', 'DAGS_FOLDER/init_ocp.py', '--cfg-path', '/tmp/tmpslxftz0m'] [2024-03-06T00:06:42.102+0800] INFO - Job 936: Subtask init_ocp_meta [2024-03-06T00:06:42.246+0800] INFO - Running on host wzserver [2024-03-06T00:06:42.416+0800] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_ocp AIRFLOW_CTX_TASK_ID=init_ocp_meta AIRFLOW_CTX_EXECUTION_DATE=2024-02-07T07:01:06.501917+00:00 AIRFLOW_CTX_TRY_NUMBER=10 AIRFLOW_CTX_DAG_RUN_ID=manual__2024-02-07T07:01:06.501917+00:00 [2024-03-06T00:06:42.465+0800] INFO - meta tenant is {'host': '172.17.2.157', 'port': 2883, 'user': 'root@metasys#OB_METADB', 'password': '*******'} monitor tenant is {'host': '172.17.2.157', 'port': 2883, 'user': 'root@metamonitor#OB_METADB', 'password': '********'} [2024-03-06T00:06:42.466+0800] INFO - try to clean database... [2024-03-06T00:06:42.469+0800] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 613, in connect sock = socket.create_connection( File "/usr/lib64/python3.9/socket.py", line 843, in create_connection raise err File "/usr/lib64/python3.9/socket.py", line 831, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 217, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_ocp.py", line 110, in init_ocp_meta common.drop_database('ocp', meta_tenant, monitor_tenant) File "/oat/task_engine/plugins/common.py", line 1087, in drop_database with pymysql.connect(**meta_tenant, charset='utf8mb4') as conn: File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 353, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 664, in connect raise exc pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '172.17.2.157' ([Errno 111] Connection refused)") [2024-03-06T00:06:42.482+0800] INFO - Marking task as FAILED. dag_id=init_ocp, task_id=init_ocp_meta, execution_date=20240207T070106, start_date=20240305T160642, end_date=20240305T160642 [2024-03-06T00:06:42.485+0800] INFO - Running statement: update oat_audit set status='failed', update_time=utc_timestamp(), failed_reason=%s where id=%s, parameters: ['failed task instance is init_ocp__init_ocp_meta__20240207 and exception information is (2003, "Can\'t connect to MySQL server on \'172.17.2.157\' ([Errno 111] Connection refused)")', 13] [2024-03-06T00:06:42.486+0800] INFO - Rows affected: 1 [2024-03-06T00:06:42.512+0800] ERROR - Failed to execute job 936 for task init_ocp_meta ((2003, "Can't connect to MySQL server on '172.17.2.157' ([Errno 111] Connection refused)"); 9672) [2024-03-06T00:06:42.548+0800] INFO - Task exited with return code 1 [2024-03-06T00:06:42.612+0800] INFO - 0 downstream tasks scheduled from follow-on schedule check