使用 Canal 从 MySQL 数据库同步数据到 OceanBase 数据库结果正在observe不可见

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】
【 使用版本 】
observer:4.0
【问题现象及影响】
使用 Canal 从 MySQL 数据库同步数据到 OceanBase 数据库(参考:https://www.oceanbase.com/docs/community-observer-cn-10000000000900964),在canal-adapter 中可以看到mysql的数据在同步给OceanBase,在observer中也可以看到同步的database,但是database里面的表在observer是空的无法同步,这是什么原因造成的表无法同步,或者是同步了无法在observer中查看
【附件】
mysql侧数据

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| db1                |
| db10               |
| db11               |
| db12               |
| db13               |
| db14               |
| db15               |
| db16               |
| db17               |
| db2                |
| db20               |
| db3                |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test_mysql_to_ob   |
+--------------------+
17 rows in set (0.00 sec)

observer 经过同步后的结果

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| db1                |
| db11               |
| db15               |
| db16               |
| db17               |
| db2                |
| db20               |
| db3                |
| information_schema |
| LBACSYS            |
| mysql              |
| oceanbase          |
| ORAAUDITOR         |
| SYS                |
| test               |
| test_mysql_to_ob   |
+--------------------+
16 rows in set (0.06 sec)

mysql侧test_mysql_to_ob的内容

mysql> show tables;
+----------------------------+
| Tables_in_test_mysql_to_ob |
+----------------------------+
| tbl1                       |
| tbl2                       |
+----------------------------+
2 rows in set (0.00 sec)

observer侧test_mysql_to_ob的内容

mysql> use test_mysql_to_ob;
Database changed
mysql> show tables;
Empty set (0.05 sec)
1 个赞

方便的话可以贴一下 logs/adapter/adapter.log 这个日志。如果 conf/application.yml 里配置的 outerAdapters 有 logger 的话,应该所有的 DDL 和 DML 都会打印有打印的,先看看里面有没有报错啥的。

1 个赞
2023-06-26 10:03:10.604 [main] INFO  o.s.c.annotation.AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@76a2ddf3: startup date [Mon Jun 26 10:03:10 UTC 2023]; root of context hierarchy
2023-06-26 10:03:10.927 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$696a24cd] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2023-06-26 10:03:11.131 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - No active profile set, falling back to default profiles: default
2023-06-26 10:03:11.153 [main] INFO  o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@7b02e036: startup date [Mon Jun 26 10:03:11 UTC 2023]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@76a2ddf3
2023-06-26 10:03:11.745 [main] INFO  org.springframework.cloud.context.scope.GenericScope - BeanFactory id=d918d7e8-5091-3baa-be10-1466c8ff3b6d
2023-06-26 10:03:11.819 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$696a24cd] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2023-06-26 10:03:12.088 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8081 (http)
2023-06-26 10:03:12.107 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8081"]
2023-06-26 10:03:12.119 [main] INFO  org.apache.catalina.core.StandardService - Starting service [Tomcat]
2023-06-26 10:03:12.120 [main] INFO  org.apache.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.29
2023-06-26 10:03:12.131 [localhost-startStop-1] INFO  org.apache.catalina.core.AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
2023-06-26 10:03:12.285 [localhost-startStop-1] INFO  o.a.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2023-06-26 10:03:12.285 [localhost-startStop-1] INFO  org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 1132 ms
2023-06-26 10:03:12.411 [localhost-startStop-1] INFO  o.s.boot.web.servlet.ServletRegistrationBean - Servlet statViewServlet mapped to [/druid/*]
2023-06-26 10:03:12.412 [localhost-startStop-1] INFO  o.s.boot.web.servlet.ServletRegistrationBean - Servlet dispatcherServlet mapped to [/]
2023-06-26 10:03:12.415 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2023-06-26 10:03:12.416 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2023-06-26 10:03:12.416 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2023-06-26 10:03:12.416 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
2023-06-26 10:03:12.679 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2023-06-26 10:03:12.895 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@7b02e036: startup date [Mon Jun 26 10:03:11 GMT+00:00 2023]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@76a2ddf3
2023-06-26 10:03:13.025 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.String)
2023-06-26 10:03:13.026 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String)
2023-06-26 10:03:13.027 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" onto public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2023-06-26 10:03:13.027 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String)
2023-06-26 10:03:13.028 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String,java.lang.String)
2023-06-26 10:03:13.028 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String)
2023-06-26 10:03:13.029 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.List<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.destinations()
2023-06-26 10:03:13.032 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2023-06-26 10:03:13.033 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2023-06-26 10:03:13.058 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2023-06-26 10:03:13.058 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2023-06-26 10:03:13.269 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2023-06-26 10:03:13.280 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2023-06-26 10:03:13.282 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2023-06-26 10:03:13.283 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'environmentManager' has been autodetected for JMX exposure
2023-06-26 10:03:13.288 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2023-06-26 10:03:13.307 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2023-06-26 10:03:13.319 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=7b02e036,type=ConfigurationPropertiesRebinder]
2023-06-26 10:03:13.336 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## syncSwitch refreshed.
2023-06-26 10:03:13.337 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2023-06-26 10:03:13.343 [main] INFO  c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /Canal_Home/adapter/plugin
2023-06-26 10:03:13.418 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed
2023-06-26 10:03:13.428 [main] INFO  c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Start loading rdb mapping config ... 
2023-06-26 10:03:13.484 [main] INFO  c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Rdb mapping config loaded
2023-06-26 10:03:13.782 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
2023-06-26 10:03:13.788 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: rdb succeed
2023-06-26 10:03:13.796 [main] INFO  c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: /Canal_Home/adapter/plugin
2023-06-26 10:03:13.821 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2023-06-26 10:03:13.821 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2023-06-26 10:03:13.821 [Thread-3] INFO  c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: example <=============
2023-06-26 10:03:13.828 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2023-06-26 10:03:13.831 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2023-06-26 10:03:13.847 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2023-06-26 10:03:13.850 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 3.736 seconds (JVM running for 4.621)
2023-06-26 10:03:13.956 [Thread-3] INFO  c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: example succeed <=============
2023-06-26 10:03:14.645 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686777374000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"create database db20","table":"","ts":1687773794528,"type":"QUERY"}
2023-06-26 10:03:14.646 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686777421000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom(id int)","table":"geom","ts":1687773794528,"type":"CREATE"}
2023-06-26 10:03:14.646 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686777642000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom1(id int)","table":"geom1","ts":1687773794528,"type":"CREATE"}
2023-06-26 10:03:14.647 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686777769000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom500(id int)","table":"geom500","ts":1687773794529,"type":"CREATE"}
2023-06-26 10:03:14.647 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686778051000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom501(id int)","table":"geom501","ts":1687773794529,"type":"CREATE"}
2023-06-26 10:03:14.647 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686778155000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom502(id int)","table":"geom502","ts":1687773794529,"type":"CREATE"}
2023-06-26 10:03:14.647 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686778906000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom503(id int)","table":"geom503","ts":1687773794529,"type":"CREATE"}
2023-06-26 10:03:14.647 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686778936000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom504(id int)","table":"geom504","ts":1687773794529,"type":"CREATE"}
2023-06-26 10:03:14.648 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1686778955000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom505(id int)","table":"geom505","ts":1687773794529,"type":"CREATE"}
2023-06-26 10:03:14.649 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"","destination":"example","es":1687760387000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"ALTER USER 'canal'@'%' IDENTIFIED WITH 'mysql_native_password' AS '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9'","table":"","ts":1687773794529,"type":"QUERY"}
2023-06-26 10:03:14.649 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1687767099000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE   geom10000(id int)","table":"geom10000","ts":1687773794529,"type":"CREATE"}
2023-06-26 10:09:48.209 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":null,"database":"db20","destination":"example","es":1687774187000,"groupId":"g1","isDdl":true,"old":null,"pkNames":null,"sql":"CREATE TABLE tbl1(col1 INT PRIMARY KEY, col2 VARCHAR(20),col3 INT)","table":"tbl1","ts":1687774188208,"type":"CREATE"}
2023-06-26 10:11:06.498 [pool-7-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"col1":1,"col2":"China","col3":86},{"col1":2,"col2":"Taiwan","col3":886},{"col1":3,"col2":"Hong Kong","col3":852},{"col1":4,"col2":"Macao","col3":853},{"col1":5,"col2":"North Korea","col3":850}],"database":"db20","destination":"example","es":1687774266000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["col1"],"sql":"","table":"tbl1","ts":1687774266491,"type":"INSERT"}

在这里可以看到对db20中增加了表和插入语句的DML,我查看了observer该database也是为空的

mysql> use db20;
Database changed
mysql> show tables;
Empty set (0.03 sec)

看起来 rdb 模块是加载成功了,但是没有实际调用 RdbSyncService,推测可能的原因是映射规则没有对应上,需要检查下:

  • conf/application.yml 和 conf/rdb/**.yml 中的 dataSourceKey、destination、outerAdapterKey 是否一致
  • dbMapping 中的 database 名是否是 db20

另外,我注意到文档里的 dbMapping 部分缩进似乎不是很对,您可能要特别注意下,需要保持原文件的缩进格式:

dbMapping:
  mirrorDb: true
  database: db20