OCP metadb and serv er time is not in-sync

liarby · 2024 年8 月 22 日 13:19

【使用环境】测试环境
【 OB or 其他组件】OCP
【使用版本】4.3.1
【问题描述】使用ocp-all-in-one-4.3.1-20240805192406.el7.x86_64.tar.gz包部署OCP失败，查看日志报错ava.lang.Exception: OCP metadb and server time is not in-sync
【复现路径】问题出现前后相关操作
【附件及日志】
Error starting ApplicationContext. To display the conditions report re-run your application with ‘debug’ enabled.
2024-08-22 09:37:50.301 INFO 20549 — [main,] c.o.o.b.spring.BootstrapRunListener : failed
2024-08-22 09:37:50.481 WARN 20549 — [main,] c.o.o.s.c.analyzer.OcpFailureAnalyzer : OCP startup check failed with cause:

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘ocpCeServerApplication’: Unsatisfied dependency expressed through constructor parameter 0; nested exception
is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘ocpStartUpChecker’: Invocation of init method failed; nested exception is java.lang.Exception: OCP metadb and serv
er time is not in-sync: db time = 2024-08-22T07:37:50.219+06:00, server time = 2024-08-22T09:37:50.233+06:00, max difference allowed is 1000 ms
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:801)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:224)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:929)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:409)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1289)
at com.oceanbase.ocp.OcpCeServerApplication.main(OcpCeServerApplication.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘ocpStartUpChecker’: Invocation of init method failed; nested exception is java.lang.Exception: OCP metadb and server time is not in-sync: db time = 2024-08-22T07:37:50.219+06:00, server time = 2024-08-22T09:37:50.233+06:00, max difference allowed is 1000 ms
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:911)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788)
… 27 common frames omitted
Caused by: java.lang.Exception: OCP metadb and server time is not in-sync: db time = 2024-08-22T07:37:50.219+06:00, server time = 2024-08-22T09:37:50.233+06:00, max difference allowed is 1000 ms
at com.oceanbase.ocp.server.common.analyzer.OcpStartUpChecker.checkTime(OcpStartUpChecker.java:109)
at com.oceanbase.ocp.server.common.analyzer.OcpStartUpChecker.init(OcpStartUpChecker.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
… 40 common frames omitted

2024-08-22 09:37:50.486 ERROR 20549 — [main,] o.s.b.d.LoggingFailureAnalysisReporter :

APPLICATION FAILED TO START

Description:

OCP application startup check failed.

Action:

Please check the stack trace above for the root cause.

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！

论坛小助手 · 2024 年8 月 22 日 13:20

从您的描述来看，您在 OCP 4.3.1 测试环境中遇到了部署失败的问题，具体的错误信息提示是 OCP metadb 和 server 时间不同步。这通常意味着集群中的时间同步出现了问题，可能会影响到数据一致性和某些操作的正确性。

在这种情况下，我们建议您先检查并确保所有节点的时间都已同步。不过，为了更全面地了解问题并进行诊断，您可以使用 obdiag 工具来收集集群的基础信息。

日志采集场景: 集群基础信息
日志采集命令: obdiag gather scene run --scene=observer.base

该命令将会收集集群的基本配置和状态信息，包括但不限于时间配置、NTP 设置等，这有助于我们进一步分析时间不同步的原因。

请您执行上述命令并将生成的日志压缩包发送给我们，以便我们能够更准确地定位问题。

为了更好地帮助您解决问题，请提供以下信息：

您是如何部署 OCP 的？使用的是 OBD 还是其他方式？
是否已经尝试过手动同步时间？如果尝试过，请提供您所使用的命令。
您是否在部署前检查过 NTP 服务的状态？如果有的话，请提供 NTP 服务的状态信息。
您能否提供部署失败的具体错误日志片段？特别是关于时间不同步的部分。
您的集群中是否有使用外部的时间同步服务？如果有，请提供相关配置信息。

具体信息可参考以下文档:

通过 OBD 白屏部署 OceanBase 集群

附上敏捷诊断工具 obdiag 使用帮助链接

liarby · 2024 年8 月 22 日 14:09

补充一下，此包在centos7上面安装部署正常，在AlmaLinux8上面就报这种错，部署OCP的时候，元数据OBserver和OCP是同一台服务器

论坛小助手 · 2024 年8 月 22 日 14:15

再向您确认几个问题:

您提到在 AlmaLinux 8 上部署时遇到问题而在 CentOS 7 上部署正常请问两者的 Java 环境配置是否完全相同
您是否已经在 AlmaLinux 8 上确认了 NTP 服务的状态和服务配置是否与 CentOS 7 保持一致
您能否提供 AlmaLinux 8 上 OCP metadb 和 server 时间的具体差异以及系统时间设置的相关配置
您在 AlmaLinux 8 上部署时是否对时间同步进行了特别的配置或者调整

旭辉 · 2024 年8 月 22 日 18:43

看起来DB time是滞后的，现在ocp_meta租户可以登陆吗？登陆查看一下
select now(),@@time_zone;

也发下obd的日志

旭辉 · 2024 年8 月 26 日 17:11

这个问题有进展吗