canal中间件发送端连接OceanBase

OceanBase
#1

【产品名称】OceanBase社区版

【产品版本】3.1.0

【问题描述】

如何通过修改canal中间件配置将发送端连接到OceanBase，由于OceanBase日志形成clog、ilog形式，canal无法如MySQL binlog进行解析

2021-11-07 08:11:10.605 [destination = example , address = /192.168.63.1:2881 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:example[java.lang.IllegalStateException: unexpected binlog format query result:[]

at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.loadBinlogFormat(MysqlConnection.java:492)

at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.getBinlogFormat(MysqlConnection.java:659)

at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.preDump(MysqlEventParser.java:93)

at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$1.run(AbstractEventParser.java:176)

at java.lang.Thread.run(Thread.java:748)

]

2 个赞
#2

配置贴出来我看看

https://www.bilibili.com/video/BV1db4y1h7XU?spm_id_from=333.999.0.0

可以参考这个教程试试看呀，示例配置在视频下方评论区

#3

OB的增量事务数据是通过liboblog对外输出的，liboblog是个c++写的动态库，暂时不提供mysql binlog格式的事务数据格式

当前canal拉取数据的模式是这样的：oblogproxy代理liboblog，获取OB增量事务日志，并通过socket将数据传递给oblogclient(java编写，按指定格式反序列化数据)，canal客户端集成oblogclient消费数据。

你们如果有自己的增量数据消费中间件或者平台，可以写代码直接调用liboblog消费数据

#4

老师，我这边采用oblogproxy链接canal deploy端，即https://www.bilibili.com/video/BV1UL4y1q7AL/这个视频

canal deploy端并不能正常链接报错，日志如下

2021-11-09 07:42:27.226 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 

2021-11-09 07:42:27.239 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^shwscap.*.*$

2021-11-09 07:42:27.239 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : 

2021-11-09 07:42:27.248 [destination = example , address = null , EventParser] ERROR c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - parse events has an error

com.alibaba.otter.canal.parse.exception.CanalParseException: illegal connection is null

2021-11-09 07:42:27.332 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....


instance.properties 配置如下，烦请帮忙查看一下

canal.instance.oceanbase.rsList = 192.168.63.1:2882:2881

canal.instance.oceanbase.username = root@shwscap#obcluster

canal.instance.oceanbase.password = ***

canal.instance.oceanbase.startTimestamp = 0

canal.instance.oceanbase.logproxy.address = 192.168.63.1:2883

canal.instance.oceanbase.logproxy.sslEnabled = true

canal.instance.oceanbase.logproxy.serverCert = ../conf/${canal.instance.destination:}/ca.crt

canal.instance.oceanbase.logproxy.clientCert = ../conf/${canal.instance.destination:}/client.crt

canal.instance.oceanbase.logproxy.clientKey = ../conf/${canal.instance.destination:}/client.key

canal.instance.oceanbase.tenant = shwscap

canal.instance.parser.excludeTenantInDbName = true

canal.instance.filter.regex = shwscap.*.*

#5

canal.instance.oceanbase.logproxy.address = 192.168.63.1:2883

你的端口号不太对啊，2883是obproxy，不是logproxy，logproxy是2983

#6

这个好像不太对，我这个并没有读到conf/example/instance.properties文件的配置，无法正常连接到logproxy

2021-11-11 11:32:21.614 [destination = example , address = null , EventParser] ERROR c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - parse events has an error

com.alibaba.otter.canal.parse.exception.CanalParseException: illegal connection is null

destination = example , address = null 这里是null，读取instance.properties失败

#7

顶一下，需要麻烦提供如下canal相关配置信息参考

/usr/local/canal/conf/canal.properties

/usr/local/canal/conf/example/instance.properties