datax 迁移mysql至oceanbase 失败

OceanBase 技术问题
#1

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】OceanBase, OCP
【 使用版本 】3.1.3-OceanBase CE
【问题描述】清晰明确描述问题
【复现路径】问题出现前后相关操作
【问题现象及影响】
启动datax没有传输数据，查看datax日志，报错如下

2022-07-08 18:30:41.261 [0-0-0-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2022-07-08 18:30:41.262 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest 
] jdbcUrl:[jdbc:mysql://146.81.20.197:54493/zyh?serverTimezone=UTC&useUnicode=true&characterEncoding=utf8&useSSL=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2022-07-08 18:30:41.265 [0-0-0-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
java.lang.ArrayIndexOutOfBoundsException: 1
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.ext.ServerConnectInfo.<init>(ServerConnectInfo.java:24) ~[oceanbasev10writer-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:92) ~[oceanbasev10writer-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210) ~[oceanbasev10writer-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2022-07-08 18:30:51.257 [job-0] INFO  StandAloneJobContainerCommunicator - Total 512 records, 270274 bytes | Speed 26.39KB/s, 51 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.001s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2022-07-08 18:30:51.257 [job-0] ERROR JobContainer - 运行scheduler 模式[standalone]出错.
2022-07-08 18:30:51.258 [job-0] ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .].  - java.lang.ArrayIndexOutOfBoundsException: 1
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.ext.ServerConnectInfo.<init>(ServerConnectInfo.java:24)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:92)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
	at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
	at java.lang.Thread.run(Thread.java:748)
 - java.lang.ArrayIndexOutOfBoundsException: 1
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.ext.ServerConnectInfo.<init>(ServerConnectInfo.java:24)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:92)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
	at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
	at java.lang.Thread.run(Thread.java:748)

	at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
	at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
java.lang.ArrayIndexOutOfBoundsException: 1
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.ext.ServerConnectInfo.<init>(ServerConnectInfo.java:24) ~[na:na]
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:92) ~[na:na]
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210) ~[na:na]
	at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
2022-07-08 18:30:51.259 [job-0] INFO  StandAloneJobContainerCommunicator - Total 512 records, 270274 bytes | Speed 263.94KB/s, 512 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.001s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2022-07-08 18:30:51.259 [job-0] ERROR Engine - 

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .].  - java.lang.ArrayIndexOutOfBoundsException: 1
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.ext.ServerConnectInfo.<init>(ServerConnectInfo.java:24)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:92)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
	at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
	at java.lang.Thread.run(Thread.java:748)
 - java.lang.ArrayIndexOutOfBoundsException: 1
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.ext.ServerConnectInfo.<init>(ServerConnectInfo.java:24)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:92)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
	at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
	at java.lang.Thread.run(Thread.java:748)

	at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40)
	at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39)
	at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99)
	at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535)
	at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119)
	at com.alibaba.datax.core.Engine.start(Engine.java:92)
	at com.alibaba.datax.core.Engine.entry(Engine.java:171)
	at com.alibaba.datax.core.Engine.main(Engine.java:204)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.ext.ServerConnectInfo.<init>(ServerConnectInfo.java:24)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:92)
	at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV

【附件】

#2

写入端-即OceanBase侧 连接信息写错 了。
给你个参考：
“connection”: [
{
“jdbcUrl”: “||dsc_ob10_dsc||obcluster:test_tenant||dsc_ob10_dsc||jdbc:oceanbase://192.168.32.111:2883/lcxtest?useLocalSessionState=true&allowBatch=true&allowMultiQueries=true&rewriteBatchedStatements=true”,
…

1 个赞
#3

罗老师从那里看的写入端是OB，我研究了半天:man_facepalming:

#4

.writer.oceanbasev10write

writer是oceanbase，所以是写入OB

#5

com.alibaba.datax.plugin.writer.oceanbasev10writer.task

1 个赞
#7

writer是OceanBase v10

#8

并没有解决

#9

2022-07-21 20:06:44.979 [0-0-0-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO person (id,title,author) VALUES(?,?,?)
2022-07-21 20:06:44.984 [0-0-0-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
java.lang.NumberFormatException: multiple points
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1914) ~[na:1.8.0_181]
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122) ~[na:1.8.0_181]
at java.lang.Float.parseFloat(Float.java:451) ~[na:1.8.0_181]
at java.lang.Float.valueOf(Float.java:416) ~[na:1.8.0_181]
at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:131) ~[oceanbasev10writer-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210) ~[oceanbasev10writer-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Exception in thread “taskGroup-0” com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .]. - java.lang.NumberFormatException: multiple points

#10

./datax.py -v

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

observer: 3.1.3
obproxy: 3.2.3

#12

json文件贴一下，看看你的reader和writer配置文件

#13 
{

    "job": {

        "setting": {

            "speed": {

                "channel": 4

            },

            "errorLimit": {

                "record": 0,

                "percentage": 0.1

            }

        },

        "content": [

            {

                "reader": {

                    "name": "mysqlreader",

                    "parameter": {

                        "username": "root",

                        "password": "xxxxxxx",

                        "column": [

                            "*"

                        ],

                        "connection": [

                            {

                                "table": [

                                    "test"

                                ],

                                "jdbcUrl": ["jdbc:mysql://172.21.xx.xx:3306/datax_test?useUnicode=true&characterEncoding=utf8"]

                            }

                        ]

                    }

                },



                "writer": {

                    "name": "oceanbasev10writer",

                    "parameter": {

                        "obWriteMode": "insert",

                        "column": [

                            "*"

                        ],

                        "preSql": [

                            "truncate table test"

                        ],

                        "connection": [

                            {

                                "jdbcUrl": "||_dsc_ob10_dsc_||oceanbase:test||_dsc_ob10_dsc_||jdbc:oceanbase://172.x.x.x:2883/datax_test",

                                "table": [

                                    "test"

                                ]

                            }

                        ],

                        "username": "root",

                        "password":"XxxX11111__",

                        "writerThreadCount":10,

                        "batchSize": 1000,

                        "memstoreThreshold": "0.9"

                    }

                }

            }

        ]

    }

}
#14

jdbc:oceanbase://xxx.x.x.x:2883/datax_test?useLocalSessionState=true&allowBatch=true&allowMultiQueries=true&rewriteBatchedStatements=true

useLocalSessionState=true&allowBatch=true&allowMultiQueries=true&rewriteBatchedStatements=true

这一串加了也没用

#16

ob配置项的 jdbcUrl 那里，
||dsc_ob10_dsc||oceanbase:test|| 其中 oceanbase是否是集群名？test是否是租户名？

#17

集群名:租户名
这个确认没问题的。

很奇怪我之前测试过datax迁移mysql至OB是没问题的，当时的OB集群是obd部署的
这次的买了几个大规格的云主机使用OCP来部署集群，发现OCP里的数据库连接串的格式有点不一样，集群名带有’:1’ 例如：
mysql -h172.0.0.1 -uroot@test#oceanbase:1 -P2883 -p

但我发现datax的配置中，如集群名带有’:1’ 启动时会报连接失败。
测试mysql 命令：
mysql -h172.0.0.1 -uroot@test#oceanbase -P2883 -p
也是可以成功连接的，于是datax的配置中去掉集群名中的’:1’，连接成功但报上诉错误。

猜测是部署的步骤有问题？可是云主机重置后，继续按文档部署依旧出现这个问题

#18

你用的就是ob的驱动？不是MySQL官方的驱动吗？

你报错日志中的url和你提供的json中的writer的url可不同，驱动使用的都不同。

#19

jdbcurl 中驱动oceanbase 和 mysql 都试过的，查看日志报错信息没有变化哒

#20 
{
    "job": {
        "setting": {
            "speed": {
                "channel": 4 
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0.1
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "tpcc",
                        "password": "********",
                        "column": [
                            "*"
                        ],
                        "connection": [
                            {
                                "table": [
                                    "bmsql_oorder"
                                ],
                                "jdbcUrl": ["jdbc:mysql://127.0.0.1:3306/tpccdb?useUnicode=true&characterEncoding=utf8"]
                            }
                        ]
                    }
                },

                "writer": {
                    "name": "oceanbasev10writer",
                    "parameter": {
                        "obWriteMode": "insert",
                        "column": [
                            "*"
                        ],
                        "preSql": [
                            "truncate table bmsql_oorder"
                        ],
                        "connection": [
                            {
                                "jdbcUrl": "||_dsc_ob10_dsc_||obdemo:oboracle||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/tpcc?useLocalSessionState=true&allowBatch=true&allowMultiQueries=true&rewriteBatchedStatements=true",
                                "table": [
                                    "bmsql_oorder"
                                ]
                            }
                        ],
                        "username": "tpcc",
                        "password":"********",
                        "writerThreadCount":10,
                        "batchSize": 1000,
                        "memstoreThreshold": "0.9"
                    }
                }
            }
        ]
    }
}

https://open.oceanbase.com/docs/observer-cn/V3.1.4/10000000000449924
使用官方文档里的配置文件，只修改相关配置信息也还是报multiple points

#21

||dsc_ob10_dsc||obdemo:oboracle||dsc_ob10_dsc||
这一部分统统去掉，只需要jdbc:oceanbase://127.0.0.1:2883/tpcc这一部分就可以了