社区版OCP机器重启以后无法访问无法连接自身MetaDB

【 使用环境 】测试环境
【 OB or 其他组件 】
oceanbase-ce-3.1.3
【 使用版本 】
ocp-3.3.0-ce
【问题描述】
ocp服务器(xx.xx.xx.102)重启以后(接管的集群也一起重启),网页无法访问,查看日志显示自身oceanbase无法连接
【复现路径】
ocp为单机所需MetaDB也在自身机器上,ocp接管另外三台机器的集群

想要测试一下ocp以及集群重启,于是将所有机器重启了一遍
重启以后ocp网页无法访问
查看ocp服务器状态,发现ocp docker容器已经运行了

[root@oceanbaseocp ocp]# docker ps
CONTAINER ID        IMAGE                                                          COMMAND                  CREATED             STATUS              PORTS               NAMES
70cd2aa6e578        reg.docker.alibaba-inc.com/oceanbase/ocp-all-in-one:3.3.0-ce   "/usr/bin/supervis..."   16 hours ago        Up 28 minutes                           ocp

查看系统进程,发现ocp相关的进程有三个ocp_obproxyd.shocp-serverocp-server-3.3.0-20220427.jar,没有observer相关进程

日志/home/admin/ocp/log/ocp/ocp-server.0.out显示无法连接MetaDB:

***************************
APPLICATION FAILED TO START
***************************

Description:

OCP application startup check failed.

Action:

Please check the stack trace above for the root cause.

Run in container
Total memory is 3145728KB
Starting ocp-server: /usr/lib/jvm/java-1.8.0/bin/java -server -XX:+UseG1GC -Xms2150m -Xmx2150m -Xss512k -XX:+PrintCommandLineFlags -XX:MetaspaceSize=1024m -XX:MaxMetaspaceSize=1024m -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/home/admin/ocp-server/bin/../log/gc.log -XX:+UseGCLogFileRotation -XX:GCLogFileSize=50M -XX:NumberOfGCLogFiles=2 -XX:ErrorFile=/home/admin/ocp-server/bin/../log/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/admin/ocp-server/bin/../log/ -Dfile.encoding=UTF-8 -jar /home/admin/ocp-server/bin/../lib/ocp-server-3.3.0-20220427.jar
-XX:CompressedClassSpaceSize=1065353216 -XX:ErrorFile=/home/admin/ocp-server/bin/../log/hs_err_pid%p.log -XX:GCLogFileSize=52428800 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/admin/ocp-server/bin/../log/ -XX:InitialHeapSize=2254438400 -XX:MaxHeapSize=2254438400 -XX:MaxMetaspaceSize=1073741824 -XX:MetaspaceSize=1073741824 -XX:NumberOfGCLogFiles=2 -XX:+PrintAdaptiveSizePolicy -XX:+PrintCommandLineFlags -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:ThreadStackSize=512 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation 

   ___                             ___
  / _ \   __   ___   __ _   _ _   | _ )  __ _   ___  ___
 | (_) | / _| / -_) / _` | | ' \  | _ \ / _` | (_-< / -_)
  \___/  \__| \___| \__,_| |_||_| |___/ \__,_| /__/ \___|

 :: Spring Boot ::         (v2.3.9.RELEASE)
 ::  OCP Server ::         (v3.3.0-20220427)

2022-07-15 10:55:17.833  INFO 18936 --- [           main] o.s.c.c.s.e.NativeEnvironmentRepository  : Adding property source: classpath:/application.properties
2022-07-15 10:55:17.932 ERROR 18936 --- [           main] c.a.o.s.analyzer.OcpFailureAnalyzer      : OCP startup check failed with cause: 

org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=xx.xx.xx.102)(port=2883)(type=master) : Socket fail to connect to host:xx.xx.xx.102, port:2883. Connection refused (Connection refused)
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:612) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:669) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:700) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
        at org.springframework.cloud.config.server.environment.JdbcEnvironmentRepository.findOne(JdbcEnvironmentRepository.java:102) ~[spring-cloud-config-server-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
        at org.springframework.cloud.config.server.environment.EnvironmentRepository.findOne(EnvironmentRepository.java:31) ~[spring-cloud-config-server-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
        at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:66) ~[spring-cloud-config-server-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
        at org.springframework.cloud.config.server.environment.EnvironmentRepositoryPropertySourceLocator.locate(EnvironmentRepositoryPropertySourceLocator.java:55) ~[spring-cloud-config-server-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
        at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:52) ~[spring-cloud-context-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
        at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:47) ~[spring-cloud-context-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
        at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:98) ~[spring-cloud-context-2.2.5.RELEASE.jar!/:2.2.5.RELEASE]
        at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626) [spring-boot-2.3.9.RELEASE.jar!/:2.3.9.RELEASE]
        at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370) [spring-boot-2.3.9.RELEASE.jar!/:2.3.9.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) [spring-boot-2.3.9.RELEASE.jar!/:2.3.9.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.9.RELEASE.jar!/:2.3.9.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.9.RELEASE.jar!/:2.3.9.RELEASE]
        at com.alipay.ocp.OcpServerApplication.main(OcpServerApplication.java:23) [classes!/:3.3.0-20220427]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [ocp-server-3.3.0-20220427.jar:3.3.0-20220427]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:107) [ocp-server-3.3.0-20220427.jar:3.3.0-20220427]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [ocp-server-3.3.0-20220427.jar:3.3.0-20220427]

【附件】

重启机器后需要手动重启下ocp对应的metaDB,你可以进入到对应的metaDB
安装目录,执行下bin/observer启动

可以了,除了要启动observer,还要启动bin/obproxy

1 个赞