flinkcdc任务只能全量同步,不能增量

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】
flinkcdc版本2.4.2
【 使用版本 】
#OB版本
[root@4e94d3b3da17 ~]# obclient -h192.168.1.47 -P2881 -uroot@sys -pocean@123
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 3221503998
Server version: OceanBase_CE 4.3.5.5 (r105000012025111711-c10174c0486c38f64a2222486986bbe15d5da0dc) (Built Nov 17 2025 12:20:08)

#logproxy 版本
[root@data-source oblogproxy]# ./bin/logproxy --version
version: 4.3.5-836e752c3cfc137ae8606584d667681fc8fa50db
【问题描述】清晰明确描述问题
flinkcdc读取ob数据,能读取到全量数据, 但是ob表新增的数据(比如c1=10的增量数据)不能读取到;
【复现路径】问题出现前后相关操作
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接(右键跳转查看):

【SOP系列 22 】——故障诊断第一步(自助诊断和诊断信息收集)
#cdc作业数据源虚拟表
create table oceanbase_table(
c1 int,
c2 string,
PRIMARY KEY(c1) NOT ENFORCED
) with (
‘connector’ = ‘oceanbase-cdc’,
‘username’ = ‘root@sys’,
‘password’ = ‘ocean@123’,
‘database-name’ = ‘db1’,
‘table-name’ = ‘testtable’,
‘hostname’ = ‘192.168.1.47’,
‘port’ = ‘2881’,
‘scan.startup.mode’ = ‘initial’,
‘tenant-name’ = ‘test’, – ai说不要用sys
‘rootserver-list’ = ‘192.168.1.47:2882:2881’,
‘logproxy.host’ = ‘192.168.1.47’,
‘logproxy.port’ = ‘2983’,
‘working-mode’ = ‘memory’ – storage memory
);

#cdc作业打印输出信息
13577 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - Start to initial table whitelist
Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is com.mysql.cj.jdbc.Driver’. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
14144 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - Pattern matched tables: [db1.testtable]
14144 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - Table list: [db1.testtable]
14144 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - Start readChangeRecords process
14164 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - LogProxyClient started
14164 [Thread-8] INFO com.oceanbase.clogproxy.client.connection.ClientStream - Try to reconnect
15606 [Thread-8] INFO com.oceanbase.clogproxy.client.connection.ClientStream - Reconnect successfully
15635 [log-proxy-client-worker-1-thread-1] INFO com.oceanbase.clogproxy.client.connection.ClientHandler - ClientId: 2.0.0.1_17488_1770035315_1_test: rootserver_list=192.168.1.47:2882:2881, cluster_id=, cluster_user=root@sys, cluster_password=, , sys_user=, sys_password=, tb_white_list=test.., tb_black_list=|, start_timestamp=0, start_timestamp_us=0, timezone=+00:00, working_mode=memory connecting LogProxy: 192.168.1.47:2983
16321 [log-proxy-client-worker-1-thread-1] INFO com.oceanbase.clogproxy.client.connection.ClientHandler - Connected to LogProxyServer, ip:172.17.0.1, version:4.3.5-836e752c3cfc137ae8606584d667681fc8fa50db
23769 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - LogProxyClient packet processing started from timestamp 1770035324
23769 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - Snapshot reading started
23769 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - Start to read snapshot from db1.testtable
+I[1, b]
+I[2, a]
+I[3, a]
+I[4, a]
+I[5, a]
+I[6, a]
+I[7, a]
+I[8, a]
+I[9, a]
23822 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - Read snapshot from db1.testtable finished
23822 [Legacy Source Thread - Source: oceanbase_table[1] → Sink: print_table[2] (1/1)#0] INFO com.ververica.cdc.connectors.oceanbase.source.OceanBaseRichSourceFunction - Snapshot reading finished

#logproxy.log部分显示有异常的日志如下:

#conf.json配置部分如下,只修改了ob_sys_username和ob_sys_password
{
“service_port”: 2983,
“encode_threadpool_size”: 8,
“encode_queue_size”: 20000,
“max_packet_bytes”: 67108864,
“record_queue_size”: 20000,
“read_timeout_us”: 2000000,
“read_fail_interval_us”: 1000000,
“read_wait_num”: 20000,
“send_timeout_us”: 2000000,
“send_fail_interval_us”: 1000000,
“check_quota_enable”: false,
“check_clog_enable”: true,
“command_timeout_s”: 10,
“log_quota_size_mb”: 5120,
“log_quota_day”: 7,
“log_gc_interval_s”: 43200,
“log_level”: 2,
“loh_flush_strategy”: 1,
“log_flush_level”: 2,
“log_flush_period_s”: 1,
“log_max_file_size_mb”: 1024,
“log_retention_h”: 360,
“oblogreader_path_retain_hour”: 168,
“oblogreader_lease_s”: 300,
“oblogreader_path”: “./run”,
“bin_path”: “./bin”,
“oblogreader_timezone_conf”: “…/…/conf/timezone_info.conf”,
“oblogreader_obcdc_path_template”: “…/…/obcdc/obcdc-%s.x-access/libobcdcaccess.so”,
“oblogreader_obcdc_ce_path_template”: “…/…/obcdc/obcdc-ce-%s.x-access/libobcdcaccess.so”,
“allow_all_tenant”: true,
“auth_user”: true,
“auth_use_rs”: false,
“auth_allow_sys_user”: true,
“ob_sys_username”: “57C7BF6DCA3A9F207697EB7CA7DED408”,
“ob_sys_password”: “F60F11392814924E0E492835B1FF9427”,
“counter_interval_s”: 2,
“metric_enable”: true,
“metric_interval_s”: 10,
“prometheus_port”: 2984,
“debug”: false,
“verbose”: false,
“verbose_packet”: false,
“verbose_record_read”: false,
“readonly”: false,
“count_record”: false,
“channel_type”: “plain”,
“tls_ca_cert_file”: “”,
“tls_cert_file”: “”,
“tls_key_file”: “”,
“tls_verify_peer”: true,
“liboblog_tls”: false,
“liboblog_tls_cert_path”: “”,
“binlog_log_bin_basename”: “./run”,

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试,在发帖时输入 [@论坛小助手] 即可召唤小助手,欢迎试用!

2 个赞

根据您的描述,您遇到了 FlinkCDC 任务在读取 OceanBase 数据时,只能获取全量数据而无法获取增量数据的问题。这可能是由于多种因素导致的,例如配置问题、版本兼容性问题或者是 OceanBase 端的某些特定设置。首先,我们可以使用 obdiag 工具来收集相关日志和信息,帮助我们更准确地定位问题。

日志采集场景: 日志归档问题

日志采集命令: obdiag gather scene run --scene=observer.log_archive

该命令将帮助我们收集与日志归档相关的所有必要信息,包括但不限于归档日志的状态、配置参数等。通过分析这些日志,我们可以判断是否存在日志归档方面的问题,这可能影响到 FlinkCDC 在读取增量数据时的表现。

在您执行上述命令并提供日志文件后,我们能够进一步分析问题的原因。此外,为了更全面地了解情况,请您回答以下问题:

  1. 您是否确认已正确配置了 FlinkCDC 来读取 OceanBase 的增量数据?请检查 FlinkCDC 的配置文件中关于增量读取的相关设置。
  2. 当前 OceanBase 集群的 Binlog 功能是否已经启用?您可以登录 OceanBase 客户端,执行 SHOW PARAMETERS LIKE '%binlog%' 查看相关配置。
  3. 能否提供一些具体的日志片段或错误信息,说明在尝试读取增量数据时遇到的具体问题?
  4. LogProxy 的配置是否进行了任何自定义调整?如果有,请提供相关的配置细节。
  5. FlinkCDC 任务在执行过程中是否有任何异常信息或警告信息?

以上信息将帮助我们更有效地定位并解决问题。谢谢您的配合。

附上敏捷诊断工具 obdiag 使用帮助链接

1 个赞

在线同步工具吗

1 个赞

通过flinkcdc connector (flink-sql-connector-oceanbase-cdc-2.4.2.jar)读取OB数据 , 现在发现不能获取到OB表的增量数据。 我现在有个猜想,是CE版不支持增量吗

1 个赞

跟踪一下