-
版本信息：
ocp: 4.3.5-20250303105901
ob: 4.2.2.1-101000012024030709 目标版本：oceanbase-ce-4.2.5.2-102010012025022610.el7.x86_64 -
备集群 升级成功。
-
主集群 3节点升级任务卡住了。
卡在任务：Execute upgrade health checker script
多次重试，都失败。也无法跳过。
日志：
subtask_9171685.log (76.1 KB)
日志最后两句最为奇怪：
2025-03-31 23:33:09.920 INFO 25034 --- [manual-subtask-executor12,a0327b7a330a68f8,7969b8d17f14f688] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,,<null>
2025-03-31 23:33:09.922 INFO 25034 --- [manual-subtask-executor12,a0327b7a330a68f8,7969b8d17f14f688] c.o.ocp.common.lang.pattern.Retry : wait for 15 seconds
2025-03-31 23:33:24.924 INFO 25034 --- [manual-subtask-executor12,a0327b7a330a68f8,7969b8d17f14f688] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.10.10.38:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=08470b2b-c8b0-40dc-9b82-9a4dc4181152), params:null
2025-03-31 23:33:24.930 INFO 25034 --- [manual-subtask-executor12,a0327b7a330a68f8,7969b8d17f14f688] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,null,,<null>
2025-03-31 23:33:24.933 INFO 25034 --- [manual-subtask-executor12,a0327b7a330a68f8,7969b8d17f14f688] c.o.o.s.t.b.c.helper.UpgradeTaskHelper : execute script upgrade_health_checker.py finished
Set state for subtask: 9171685, operation:RETRY, state: SUCCESSFUL
Set state for subtask: 9171685, operation:RETRY, state: FAILED
先是说 SUCCESSFUL， 后说 FAILED 。怀疑是这个版本 OCP 问题。
查看 ocp.log
发现下面日志：
2025-03-31 23:51:15.370 WARN 2696 --- [manual-subtask-executor1,a0327b7a330a68f8,9444d552e6180363] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 1205, SQLState: HY000
2025-03-31 23:51:15.370 ERROR 2696 --- [manual-subtask-executor1,a0327b7a330a68f8,9444d552e6180363] o.h.engine.jdbc.spi.SqlExceptionHelper : (conn=1051017) Lock wait timeout exceeded; try restarting transaction
2025-03-31 23:51:15.383 WARN 2696 --- [manual-subtask-executor1,a0327b7a330a68f8,9444d552e6180363] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed.
org.springframework.dao.PessimisticLockingFailureException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.PessimisticLockException: could not execute statement
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:271)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:243)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
at com.sun.proxy.$Proxy426.updateStateByIdIn(Unknown Source)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskFinishedProcessor.lambda$handleExecuteTask$3(SubtaskFinishedProcessor.java:84)
at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskFinishedProcessor.transactionWithoutResult(SubtaskFinishedProcessor.java:139)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskFinishedProcessor.handleExecuteTask(SubtaskFinishedProcessor.java:84)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskFinishedProcessor.markRelatedSubtask(SubtaskFinishedProcessor.java:40)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskAccessor.setExecuteEnded(SubtaskAccessor.java:134)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskAccessor$$FastClassBySpringCGLIB$$5fd21405.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskAccessor$$EnhancerBySpringCGLIB$$5facfbef.setExecuteEnded(<generated>)
at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:143)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.hibernate.PessimisticLockException: could not execute statement
at org.hibernate.dialect.MySQLDialect$3.convert(MySQLDialect.java:547)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:200)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:80)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:50)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:458)
at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:376)
at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1483)
