【 使用环境 】 测试环境
【 OB or 其他组件 】源端mysql 同步数据通过canal 目标端oceanbase
【 使用版本 】
mysql5.7.31 canal-1.1.6 oceanbase4.2
【问题描述】
源端mysql通过canal同步数据到oceanbase时，canal-adapter报错，报错时，mysql，canal deployer
日志均无报错，canal_deployer和canal_adapter安装在同一台服务器，且mysql到canal，canal到OB无出现网络问题。
报错信息如下
2023-08-30 16:17:51.476 [Thread-5] ERROR c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - com.alibaba.otter.canal.protocol.exception.CanalClientException: deserializer failed Error sync but ACK!
当时mysql在做的SQL为简单的批量insert语句，约50000条，报错时,OB只同步到了1720条
【问题现象及影响】
报错后，同步数据中断
【附件】
同步的表结构如下
CREATE TABLE test1.sbtest1 (
id int(11) NOT NULL AUTO_INCREMENT,
k int(11) NOT NULL DEFAULT ‘0’,
c char(120) NOT NULL DEFAULT ‘’,
pad char(60) NOT NULL DEFAULT ‘’,
PRIMARY KEY (id),
KEY k_1 (k)
);
报错时，canal_deployer的日志如下
2023-08-30 16:17:21.755 [destination = example , address = /192.168.167.104:3310 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - —> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000021,position=4,serverId=2,gtid=,timestamp=1693383369000] cost : 437ms , the next step is binlog dump
canal-adapter日志如下
2023-08-30 16:17:51.476 [Thread-5] ERROR c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - com.alibaba.otter.canal.protocol.exception.CanalClientException: deserializer failed Error sync but ACK!
ob日志如附件
observer.log.wf.7z (392 字节)
canal deployer参数如下
canal.instance.mysql.slaveId=1301
enable gtid use true/false
canal.instance.gtidon=false
position info
canal.instance.master.address=192.168.167.104:3310
canal.instance.master.journal.name=mysql-bin.000021
canal.instance.master.position=190
canal.instance.master.timestamp=
canal.instance.master.gtid=
rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=
table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=
username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=111111
canal-adapter 参数如下
application.yml
server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
mode: tcp #tcp kafka rocketMQ rabbitMQ
flatMessage: true
zookeeperHosts:
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
consumerProperties:
# canal tcp consumer
canal.tcp.server.host: 127.0.0.1:11111
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
srcDataSources:
test1DS:
url: jdbc:mysql://192.168.167.104:3310/test1?useUnicode=true #数据库
username: root
password: 111111
canalAdapters:
- instance: example # canal instance Name or mq topic name
groups:- groupId: g1
outerAdapters:- name: logger
- name: rdb
key: mysql1
properties:
jdbc.driverClassName: com.mysql.jdbc.Driver
jdbc.url: jdbc:mysql://192.168.167.127:2881/test1?useUnicode=true
jdbc.username: root@sys
jdbc.password: 111111
threads: 1
commitSize: 500
- groupId: g1
rdb文件配置如下
dataSourceKey: test1DS
destination: example
groupId: g1
outerAdapterKey: mysql1
concurrent: true
dbMapping:
mirrorDb: true
database: test1