[2023-06-27T17:13:58.421+0800] INFO - Dependencies all met for [2023-06-27T17:13:58.440+0800] INFO - Dependencies all met for [2023-06-27T17:13:58.440+0800] INFO - -------------------------------------------------------------------------------- [2023-06-27T17:13:58.440+0800] INFO - Starting attempt 1 of 1 [2023-06-27T17:13:58.440+0800] INFO - -------------------------------------------------------------------------------- [2023-06-27T17:13:58.464+0800] INFO - Executing on 2023-06-27 09:00:46.028868+00:00 [2023-06-27T17:13:58.468+0800] INFO - Started process 23674 to run task [2023-06-27T17:13:58.474+0800] INFO - Running: ['airflow', 'tasks', 'run', 'init_ocp', 'start_ocp_container', 'manual__2023-06-27T09:00:46.028868+00:00', '--job-id', '52', '--raw', '--subdir', 'DAGS_FOLDER/init_ocp.py', '--cfg-path', '/tmp/tmpc0fm2mlv', '--map-index', '0'] [2023-06-27T17:13:58.477+0800] INFO - Job 52: Subtask start_ocp_container [2023-06-27T17:13:58.580+0800] INFO - Running on host yqoat [2023-06-27T17:13:58.728+0800] INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=init_ocp AIRFLOW_CTX_TASK_ID=start_ocp_container AIRFLOW_CTX_EXECUTION_DATE=2023-06-27T09:00:46.028868+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2023-06-27T09:00:46.028868+00:00 [2023-06-27T17:13:58.752+0800] INFO - start run ocp container on server 2 [2023-06-27T17:13:58.753+0800] INFO - Running statement: select oat_server.id, oat_credential.id as credential_id, ip, ssh_port, username, password, auth_type, key_data, passphrase from oat_server, oat_credential where oat_server.credential_id=oat_credential.id and oat_server.id=%s, parameters: [2] [2023-06-27T17:13:58.755+0800] INFO - Rows affected: 1 [2023-06-27T17:13:58.756+0800] INFO - try to clean environment... [2023-06-27T17:13:58.771+0800] INFO - Connected (version 2.0, client OpenSSH_7.4) [2023-06-27T17:13:58.856+0800] INFO - Authentication (password) successful! [2023-06-27T17:13:58.857+0800] INFO - execute command on 192.168.0.68: if docker ps -a --format '{{.Names}}' | grep -Eq '^ocp$'; then docker rm -f ocp; fi [2023-06-27T17:13:59.017+0800] INFO - execute command on 192.168.0.68: exit_code=0 for port in 8080 do if ss -tln | grep ":$port "; then echo "Port $port on 192.168.0.68 is already used! " exit_code=1 fi done exit $exit_code [2023-06-27T17:13:59.120+0800] INFO - execute command on 192.168.0.68: mkdir -p /home/admin/logs/ocp/{ocp,obproxy/log,obproxy/minidump} && chown -R 500:500 /home/admin/logs/ocp docker run -d --name ocp --net=host --ulimit nofile=655360:655360 --cpu-period 100000 \ --cpu-quota 800000 --memory=18G \ -e OCP_METADB_HOST=192.168.0.68 -e OCP_METADB_PORT=2883 -e OCP_METADB_USER=root@ocp_meta#meta_cluster \ -e OCP_METADB_PASSWORD=admin -e OCP_METADB_DBNAME=ocp \ -v /home/admin/logs/ocp:/home/admin/logs --restart on-failure:5 reg.docker.alibaba-inc.com/oceanbase/ocp-all-in-one:3.3.4-20221202162515 || exit 1 for i in {1..60}; do sleep 8 if ! docker ps --format "{{.Names}}" | grep -qw ocp; then echo "ocp container is not running"; exit 1 fi if curl -Ls --connect-timeout 3 "127.0.0.1:8080/services?Action=GetObProxyConfig&User_ID=admin&UID=alibaba" | grep successful; then echo "container is ready"; exit fi echo "not ready, try again..." done echo "ocp container is not ready in 8 mins, please check" && exit 1 [2023-06-27T17:13:59.346+0800] INFO - f9342bb3e7d2929458b23b807a5df44b0423b53555b60c40d7d72973a42e812c [2023-06-27T17:14:07.639+0800] INFO - not ready, try again... [2023-06-27T17:14:15.696+0800] INFO - not ready, try again... [2023-06-27T17:14:23.901+0800] INFO - not ready, try again... [2023-06-27T17:14:31.950+0800] INFO - not ready, try again... [2023-06-27T17:14:40.002+0800] INFO - not ready, try again... [2023-06-27T17:14:48.059+0800] INFO - not ready, try again... [2023-06-27T17:14:56.109+0800] INFO - not ready, try again... [2023-06-27T17:15:04.163+0800] INFO - not ready, try again... [2023-06-27T17:15:12.215+0800] INFO - not ready, try again... [2023-06-27T17:15:20.272+0800] INFO - not ready, try again... [2023-06-27T17:15:28.325+0800] INFO - not ready, try again... [2023-06-27T17:15:36.374+0800] INFO - not ready, try again... [2023-06-27T17:15:44.427+0800] INFO - not ready, try again... [2023-06-27T17:15:52.487+0800] INFO - not ready, try again... [2023-06-27T17:16:00.550+0800] INFO - not ready, try again... [2023-06-27T17:16:08.634+0800] INFO - not ready, try again... [2023-06-27T17:16:16.693+0800] INFO - not ready, try again... [2023-06-27T17:16:24.828+0800] INFO - not ready, try again... [2023-06-27T17:16:32.886+0800] INFO - not ready, try again... [2023-06-27T17:16:40.936+0800] INFO - not ready, try again... [2023-06-27T17:16:48.987+0800] INFO - not ready, try again... [2023-06-27T17:16:57.054+0800] INFO - not ready, try again... [2023-06-27T17:17:05.110+0800] INFO - not ready, try again... [2023-06-27T17:17:13.178+0800] INFO - not ready, try again... [2023-06-27T17:17:21.236+0800] INFO - not ready, try again... [2023-06-27T17:17:29.289+0800] INFO - not ready, try again... [2023-06-27T17:17:37.348+0800] INFO - not ready, try again... [2023-06-27T17:17:45.404+0800] INFO - not ready, try again... [2023-06-27T17:17:53.472+0800] INFO - not ready, try again... [2023-06-27T17:18:01.536+0800] INFO - not ready, try again... [2023-06-27T17:18:09.593+0800] INFO - not ready, try again... [2023-06-27T17:18:17.646+0800] INFO - not ready, try again... [2023-06-27T17:18:25.728+0800] INFO - not ready, try again... [2023-06-27T17:18:33.785+0800] INFO - not ready, try again... [2023-06-27T17:18:41.839+0800] INFO - not ready, try again... [2023-06-27T17:18:49.898+0800] INFO - not ready, try again... [2023-06-27T17:18:57.951+0800] INFO - not ready, try again... [2023-06-27T17:19:06.008+0800] INFO - not ready, try again... [2023-06-27T17:19:14.061+0800] INFO - not ready, try again... [2023-06-27T17:19:22.120+0800] INFO - not ready, try again... [2023-06-27T17:19:30.176+0800] INFO - not ready, try again... [2023-06-27T17:19:38.238+0800] INFO - not ready, try again... [2023-06-27T17:19:46.290+0800] INFO - not ready, try again... [2023-06-27T17:19:54.344+0800] INFO - not ready, try again... [2023-06-27T17:20:02.401+0800] INFO - not ready, try again... [2023-06-27T17:20:10.471+0800] INFO - not ready, try again... [2023-06-27T17:20:18.520+0800] INFO - not ready, try again... [2023-06-27T17:20:26.694+0800] INFO - not ready, try again... [2023-06-27T17:20:34.781+0800] INFO - not ready, try again... [2023-06-27T17:20:42.828+0800] INFO - not ready, try again... [2023-06-27T17:20:50.884+0800] INFO - not ready, try again... [2023-06-27T17:20:58.936+0800] INFO - not ready, try again... [2023-06-27T17:21:06.988+0800] INFO - not ready, try again... [2023-06-27T17:21:15.041+0800] INFO - not ready, try again... [2023-06-27T17:21:23.094+0800] INFO - not ready, try again... [2023-06-27T17:21:31.144+0800] INFO - not ready, try again... [2023-06-27T17:21:39.192+0800] INFO - not ready, try again... [2023-06-27T17:21:47.263+0800] INFO - not ready, try again... [2023-06-27T17:21:55.321+0800] INFO - not ready, try again... [2023-06-27T17:22:03.374+0800] INFO - not ready, try again... [2023-06-27T17:22:03.374+0800] INFO - ocp container is not ready in 8 mins, please check [2023-06-27T17:22:03.376+0800] ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/decorators/base.py", line 217, in execute return_value = super().execute(context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/oat/task_engine/dags/init_ocp.py", line 244, in start_ocp_container raise RuntimeError('start ocp container failed') RuntimeError: start ocp container failed [2023-06-27T17:22:03.509+0800] INFO - Marking task as FAILED. dag_id=init_ocp, task_id=start_ocp_container, map_index=0, execution_date=20230627T090046, start_date=20230627T091358, end_date=20230627T092203 [2023-06-27T17:22:03.511+0800] INFO - Running statement: update oat_audit set status='failed', update_time=utc_timestamp(), failed_reason=%s where id=%s, parameters: ['failed task instance is init_ocp__start_ocp_container__20230627 and exception information is start ocp container failed', 13] [2023-06-27T17:22:03.513+0800] INFO - Rows affected: 1 [2023-06-27T17:22:03.536+0800] ERROR - Failed to execute job 52 for task start_ocp_container (start ocp container failed; 23674) [2023-06-27T17:22:03.556+0800] INFO - Task exited with return code 1 [2023-06-27T17:22:03.594+0800] INFO - 0 downstream tasks scheduled from follow-on schedule check