ocp 报错com.oceanbase.ocp.core.distributed.lock.LockException: Failed to lock mutex at AlarmSendNotificationdoNotify

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】ocp421
【问题描述】清晰明确描述问题 ocp无法访问
【附件及日志】

2024-10-08 09:34:24.766 WARN 246 — [SimpleAsyncTaskExecutor-24967458,] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: null
2024-10-08 09:34:24.766 ERROR 246 — [SimpleAsyncTaskExecutor-24967458,] o.h.engine.jdbc.spi.SqlExceptionHelper : wait millis 2000, active 200, maxActive 200, creating 0
2024-10-08 09:34:24.767 ERROR 246 — [SimpleAsyncTaskExecutor-24967458,] .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected exception occurred invoking async method: public void com.oceanbase.ocp.alarm.schedule.AlarmNotifier.doNotifyTask()

com.oceanbase.ocp.core.distributed.lock.LockException: Failed to lock mutex at AlarmSendNotificationdoNotify
at com.oceanbase.ocp.core.distributed.lock.jdbc.JdbcLockRegistry$JdbcLock.rethrowAsLockException(JdbcLockRegistry.java:271)
at com.oceanbase.ocp.core.distributed.lock.jdbc.JdbcLockRegistry$JdbcLock.tryLock(JdbcLockRegistry.java:243)
at com.oceanbase.ocp.core.distributed.lock.jdbc.JdbcLockRegistry$JdbcLock.tryLock(JdbcLockRegistry.java:222)
at com.oceanbase.ocp.alarm.schedule.AlarmNotifier.lockAndExec(AlarmNotifier.java:101)
at com.oceanbase.ocp.alarm.schedule.AlarmNotifier.doNotifyTask(AlarmNotifier.java:137)
at com.oceanbase.ocp.alarm.schedule.AlarmNotifier$$FastClassBySpringCGLIB$$ca38577b.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67)
at com.oceanbase.ocp.core.distributed.lock.jdbc.JdbcLockRepository.deleteExpired(JdbcLockRepository.java:117)
at com.oceanbase.ocp.core.distributed.lock.jdbc.JdbcLockRepository.acquire(JdbcLockRepository.java:89)
at com.oceanbase.ocp.core.distributed.lock.jdbc.JdbcLockRegistry$JdbcLock.doLock(JdbcLockRegistry.java:263)
at com.oceanbase.ocp.core.distributed.lock.jdbc.JdbcLockRegistry$JdbcLock.tryLock(JdbcLockRegistry.java:236)
… 11 common frames omitted
Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:276)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:284)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:164)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421)
… 19 common frames omitted
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 2000, active 200, maxActive 200, creating 0
at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1775)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1427)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1407)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1397)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
… 26 common frames omitted

1.麻烦描述下做什么操作报出来的这个错误
2.发下ocp_meta 和 ocp_monitor租户的配置
3.发下ocp-server.log

1.国庆前关闭服务器后,再次启动就报这个错了
2.metaDb配置

,目前ocp无法启动
3.ocp日志
log.zip (3.7 MB)

应该是因为服务器不正常关闭的操作导致的,没有直接去正常docker stop

我联系ocp老师看下,有进展回复你

好的,谢谢

sys租户执行如下sql --查看租户连接数

select TENANT,USER,count(0) from  GV$OB_PROCESSLIST group by TENANT,USER order by TENANT,USER;
1 个赞

结果如下

ocp_meta租户是使用的Oracle模式吗?

是的

修改ocp_meta租户的meta_database库的表config_properties的key为ocp.host.remote-command.min-timeout-millis的 这个配置项。
将这个配置项的值改为小于30s,推荐20s,试下

update config_properties set default_value=20000 where `key`='ocp.host.remote-command.min-timeout-millis';

这个问题解决了吗?