2024-04-16 11:09:08.847 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=18790, context=Context{parallelIdx=-1, stringMap={ob_tenant_id=1002, tenant_id=7, checkBeforeDataBackup=true, tenant_name=myoadb, ob_version=4.3.0.1, startClusterCompact=false, task_instance_id=16693, task_operation=execute, ob_cluster_id=1713144534, startLogBackup=true, backupDim=TENANT, cluster_id=4, storageBaseInfo=******, tenantName=myoadb, backupMode=physical, triggerType=BACKUP_NOW, latest_execution_start_time=2024-04-16T11:09:08.821+08:00, sub_task_instance_name=Start data backup, backupNow=true, sub_task_instance_id=18790, dataBackupMode=FULL_BACKUP, obClusterName=myob}, listMap={}}, executor=xx.xx.xx.75 2024-04-16 11:09:08.867 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myob, obClusterId=1713144534, tenantName=sys, username=root, address=xx.xx.xx.11, port=2881, database=oceanbase) 2024-04-16 11:09:08.870 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myob, obClusterId=1713144534, tenantName=sys, username=root, address=xx.xx.xx.15, port=2881, database=oceanbase) 2024-04-16 11:09:08.873 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:xx.xx.xx.14, sql port:2881 2024-04-16 11:09:08.875 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myob, obClusterId=1713144534, tenantName=sys, username=root, address=xx.xx.xx.11, port=2881, database=oceanbase) 2024-04-16 11:09:08.877 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myob, obClusterId=1713144534, tenantName=sys, username=root, address=xx.xx.xx.15, port=2881, database=oceanbase) 2024-04-16 11:09:08.880 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:xx.xx.xx.14, sql port:2881 2024-04-16 11:09:08.883 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-04-16 11:09:08.886 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-04-16 11:09:08.909 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] .o.b.i.o.PhysicalBackupOpsServiceFactory : start tenant full data backup, clusterId=4, tenantName=myoadb 2024-04-16 11:09:08.917 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myob, tenantName=sys, dbUser=root 2024-04-16 11:09:08.920 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myob, obClusterId=1713144534, tenantName=sys, username=root, address=xx.xx.xx.11, port=2881, database=oceanbase) 2024-04-16 11:09:08.922 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myob, obClusterId=1713144534, tenantName=sys, username=root, address=xx.xx.xx.15, port=2881, database=oceanbase) 2024-04-16 11:09:08.925 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:xx.xx.xx.14, sql port:2881 2024-04-16 11:09:08.930 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myob, obClusterId=1713144534, tenantName=sys, username=root, address=xx.xx.xx.11, port=2881, database=oceanbase) 2024-04-16 11:09:08.932 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myob, obClusterId=1713144534, tenantName=sys, username=root, address=xx.xx.xx.15, port=2881, database=oceanbase) 2024-04-16 11:09:08.935 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ObConnectors : [obsdk]:connected server ip:xx.xx.xx.14, sql port:2881 2024-04-16 11:09:08.937 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-04-16 11:09:08.940 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [300000000] 2024-04-16 11:09:08.943 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter system backup tenant = ? description = ?, args: [myoadb, ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2024-04-16 11:09:08.0] 2024-04-16 11:09:08.949 WARN 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] update failed, sql:[alter system backup tenant = ? description = ?], error message:[PreparedStatementCallback; SQL [alter system backup tenant = ? description = ?]; (conn=3222091858) backup can not start, because log archive status is not doing.; nested exception is java.sql.SQLTransientConnectionException: (conn=3222091858) backup can not start, because log archive status is not doing.] 2024-04-16 11:09:08.952 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[Y10B42C0A8160E-0006161945936F39-0-0] 2024-04-16 11:09:08.954 INFO 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-04-16 11:09:08.957 ERROR 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] .o.b.i.o.PhysicalBackupOpsServiceFactory : start tenant:myoadb full data backup failed, ex=SQL [alter system backup tenant = ? description = ?; args:myoadb,ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2024-04-16 11:09:08.0]; SQL state [HY000]; error code [9040]; traceId [Y10B42C0A8160E-0006161945936F39-0-0]; message [(conn=3222091858) backup can not start, because log archive status is not doing.] com.oceanbase.ocp.obsdk.exception.OceanBaseException: (conn=3222091858) backup can not start, because log archive status is not doing. at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.customizeException(ConnectTemplate.java:433) at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.customizeException(ConnectTemplate.java:422) at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.updateInner(ConnectTemplate.java:298) at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.update(ConnectTemplate.java:283) at com.oceanbase.backup.core.obsdk.operator.MysqlPhysicalTenantWithSysBackupOperator.startFullBackup(MysqlPhysicalTenantWithSysBackupOperator.java:175) at com.oceanbase.ocp.backup.internal.operation.PhysicalBackupOpsServiceFactory.startFullDataBackup(PhysicalBackupOpsServiceFactory.java:253) at com.oceanbase.ocp.backup.internal.operation.PhysicalBackupOpsServiceFactory.startDataBackup(PhysicalBackupOpsServiceFactory.java:237) at com.oceanbase.ocp.backup.internal.operation.PhysicalBackupOpsServiceFactory.startDataBackup(PhysicalBackupOpsServiceFactory.java:465) at com.oceanbase.ocp.backup.internal.operation.PhysicalBackupTaskService.triggerDataBackup(PhysicalBackupTaskService.java:254) at com.oceanbase.ocp.backup.internal.task.schedule.StartDataBackupTask.run(StartDataBackupTask.java:82) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core. task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:203) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:197) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:134) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLTransientConnectionException: (conn=3222091858) backup can not start, because log archive status is not doing. at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:315) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:243) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:161) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeUpdate(JDBC4PreparedStatement.java:195) at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeUpdate(DruidPooledPreparedStatement.java:255) at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:973) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:656) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:968) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1023) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1033) at com.oceanbase.ocp.obsdk.connec tor.ConnectTemplate.updateInner(ConnectTemplate.java:294) ... 18 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: backup can not start, because log archive status is not doing. at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:199) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:180) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:372) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:234) ... 27 common frames omitted Caused by: java.sql.SQLException: backup can not start, because log archive status is not doing. at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2192) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2057) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1951) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:370) ... 28 common frames omitted 2024-04-16 11:09:08.965 ERROR 18779 --- [pool-manual-subtask-executor13,1916131e78d04ea6,7b4c45562bd6] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Failed to start data backup. Error message: (conn=3222091858) backup can not start, because log archive status is not doing.. com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_START_DATA_BACKUP_FAILED, args=(conn=3222091858) backup can not start, because log archive status is not doing. at com.oceanbase.ocp.backup.internal.operation.PhysicalBackupOpsServiceFactory.startFullDataBackup(PhysicalBackupOpsServiceFactory.java:256) at com.oceanbase.ocp.backup.internal.operation.PhysicalBackupOpsServiceFactory.startDataBackup(PhysicalBackupOpsServiceFactory.java:237) at com.oceanbase.ocp.backup.internal.operation.PhysicalBackupOpsServiceFactory.startDataBackup(PhysicalBackupOpsServiceFactory.java:465) at com.oceanbase.ocp.backup.internal.operation.PhysicalBackupTaskService.triggerDataBackup(PhysicalBackupTaskService.java:254) at com.oceanbase.ocp.backup.internal.task.schedule.StartDataBackupTask.run(StartDataBackupTask.java:82) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:203) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:197) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:134) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto r.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Set state for subtask: 18790, operation:EXECUTE, state: FAILED