【 使用环境 】 测试环境
【 OB or 其他组件 】datax
【 使用版本 】OB：4.2.2， mysql：5.7
OB：
MySQL [test]> select version();
±-----------------------------+
| version() |
±-----------------------------+
| 5.7.25-OceanBase_CE-v4.2.2.0 |
±-----------------------------+
MYSQL：
MySQL [tpcc]> select version();
±----------+
| version() |
±----------+
| 5.7.35 |
±----------+
【问题描述】datax迁移数据报错：ERROR JobContainer - 运行scheduler 模式[standalone]出错
【复现路径】问题出现前后相关操作
【附件及日志】
json文件：
[root@localhost ~]# cat mysql2ob.json
{
"job": {
"setting": {
"speed": {
"channel": 4
},
"errorLimit": {
"record": 0,
"percentage": 0.1
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "root",
"column": ["*"],
"connection": [
{
"table": ["customer"],
"jdbcUrl": ["jdbc:mysql://192.168.56.103:3306/tpcc?useUnicode=true&characterEncoding=utf8"]
}
]
}
},
"writer": {
"name": "oceanbasev10writer",
"parameter": {
"obWriteMode": "insert",
"column": ["*"],
"preSql": ["truncate table customer"],
"connection": [
{
"jdbcUrl": "jdbc:oceanbase://127.0.0.1:2883/tpcc?",
"table": ["customer"]
}
],
"username": "root",
"password":"rootroot",
"writerThreadCount":10,
"batchSize": 1000,
"memstoreThreshold": "0.9"
}
}
}
]
}
}
报错：
[root@localhost 2024-03-13]# cat _root_mysql2ob_json-23_18_28.231.log
2024-03-13 23:18:28.552 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-03-13 23:18:28.557 [main] INFO Engine - the machine info =>
osInfo: Red Hat, Inc. 1.8 25.402-b06
jvmInfo: Linux amd64 3.10.0-1160.el7.x86_64
cpu num: 1
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [Copy, MarkSweepCompact]
MEMORY_NAME | allocation_size | init_size
Eden Space | 273.06MB | 273.06MB
Code Cache | 240.00MB | 2.44MB
Survivor Space | 34.13MB | 34.13MB
Compressed Class Space | 1,024.00MB | 0.00MB
Metaspace | -0.00MB | 0.00MB
Tenured Gen | 682.69MB | 682.69MB
2024-03-13 23:18:28.578 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"mysqlreader",
"parameter":{
"column":[
"*"
],
"connection":[
{
"jdbcUrl":[
"jdbc:mysql://192.168.56.103:3306/tpcc?useUnicode=true&characterEncoding=utf8"
],
"table":[
"customer"
]
}
],
"password":"****",
"username":"root"
}
},
"writer":{
"name":"oceanbasev10writer",
"parameter":{
"batchSize":1000,
"column":[
"*"
],
"connection":[
{
"jdbcUrl":"jdbc:oceanbase://127.0.0.1:2883/tpcc?",
"table":[
"customer"
]
}
],
"memstoreThreshold":"0.9",
"obWriteMode":"insert",
"password":"********",
"preSql":[
"truncate table customer"
],
"username":"root",
"writerThreadCount":10
}
}
}
],
"setting":{
"errorLimit":{
"percentage":0.1,
"record":0
},
"speed":{
"channel":4
}
}
}
2024-03-13 23:18:28.595 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2024-03-13 23:18:28.596 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2024-03-13 23:18:28.596 [main] INFO JobContainer - DataX jobContainer starts job.
2024-03-13 23:18:28.598 [main] INFO JobContainer - Set jobId = 0
2024-03-13 23:18:29.004 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.56.103:3306/tpcc?useUnicode=true&characterEncoding=utf8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2024-03-13 23:18:29.005 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2024-03-13 23:18:29.312 [job-0] INFO DbUtils - value for query [SHOW VARIABLES LIKE 'ob_compatibility_mode'] is [MYSQL]
2024-03-13 23:18:29.341 [job-0] INFO OriginalConfPretreatmentUtil - table:[customer] all columns:[
c_id,c_d_id,c_w_id,c_first,c_middle,c_last,c_street_1,c_street_2,c_city,c_state,c_zip,c_phone,c_since,c_credit,c_credit_lim,c_discount,c_balance,c_ytd_payment,c_payment_cnt,c_delivery_cnt,c_data
].
2024-03-13 23:18:29.341 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2024-03-13 23:18:29.342 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (c_id,c_d_id,c_w_id,c_first,c_middle,c_last,c_street_1,c_street_2,c_city,c_state,c_zip,c_phone,c_since,c_credit,c_credit_lim,c_discount,c_balance,c_ytd_payment,c_payment_cnt,c_delivery_cnt,c_data) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
], which jdbcUrl like:[jdbc:oceanbase://127.0.0.1:2883/tpcc?&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2024-03-13 23:18:29.342 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2024-03-13 23:18:29.342 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2024-03-13 23:18:29.342 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] do prepare work .
2024-03-13 23:18:29.357 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[truncate table customer]. context info:jdbc:oceanbase://127.0.0.1:2883/tpcc?&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2024-03-13 23:18:29.445 [job-0] INFO DbUtils - value for query [show variables like 'version'] is [5.7.25-OceanBase_CE-v4.2.2.0]
2024-03-13 23:18:29.446 [job-0] INFO JobContainer - jobContainer starts to do split ...
2024-03-13 23:18:29.446 [job-0] INFO JobContainer - Job set Channel-Number to 4 channels.
2024-03-13 23:18:29.448 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
2024-03-13 23:18:29.449 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] splits to [1] tasks.
2024-03-13 23:18:29.474 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2024-03-13 23:18:29.486 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2024-03-13 23:18:29.488 [job-0] INFO JobContainer - Running by standalone Mode.
2024-03-13 23:18:29.513 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2024-03-13 23:18:29.518 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2024-03-13 23:18:29.518 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2024-03-13 23:18:29.544 [0-0-0-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2024-03-13 23:18:29.545 [0-0-0-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
2024-03-13 23:18:29.563 [0-0-0-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2024-03-13 23:18:29.572 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2024-03-13 23:18:29.576 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from customer
] jdbcUrl:[jdbc:mysql://192.168.56.103:3306/tpcc?useUnicode=true&characterEncoding=utf8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2024-03-13 23:18:29.587 [0-0-0-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2024-03-13 23:18:29.587 [0-0-0-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO customer (c_id,c_d_id,c_w_id,c_first,c_middle,c_last,c_street_1,c_street_2,c_city,c_state,c_zip,c_phone,c_since,c_credit,c_credit_lim,c_discount,c_balance,c_ytd_payment,c_payment_cnt,c_delivery_cnt,c_data) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
2024-03-13 23:18:29.590 [0-0-0-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
java.lang.NumberFormatException: multiple points
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1914) ~[na:1.8.0_402]
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122) ~[na:1.8.0_402]
at java.lang.Float.parseFloat(Float.java:451) ~[na:1.8.0_402]
at java.lang.Float.valueOf(Float.java:416) ~[na:1.8.0_402]
at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:131) ~[oceanbasev10writer-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210) ~[oceanbasev10writer-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:750) [na:1.8.0_402]
2024-03-13 23:18:39.512 [job-0] INFO StandAloneJobContainerCommunicator - Total 224 records, 118884 bytes | Speed 11.61KB/s, 22 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-03-13 23:18:39.513 [job-0] ERROR JobContainer - 运行scheduler 模式[standalone]出错.
2024-03-13 23:18:39.514 [job-0] ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .]. - java.lang.NumberFormatException: multiple points
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1914)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at java.lang.Float.valueOf(Float.java:416)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:131)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
at java.lang.Thread.run(Thread.java:750)
- java.lang.NumberFormatException: multiple points
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1914)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at java.lang.Float.valueOf(Float.java:416)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:131)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
at java.lang.Thread.run(Thread.java:750)
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
java.lang.NumberFormatException: multiple points
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1914) ~[na:1.8.0_402]
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122) ~[na:1.8.0_402]
at java.lang.Float.parseFloat(Float.java:451) ~[na:1.8.0_402]
at java.lang.Float.valueOf(Float.java:416) ~[na:1.8.0_402]
at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:131) ~[na:na]
at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210) ~[na:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_402]
2024-03-13 23:18:39.514 [job-0] INFO StandAloneJobContainerCommunicator - Total 224 records, 118884 bytes | Speed 116.10KB/s, 224 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-03-13 23:18:39.515 [job-0] ERROR Engine -
经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .]. - java.lang.NumberFormatException: multiple points
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1914)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at java.lang.Float.valueOf(Float.java:416)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:131)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
at java.lang.Thread.run(Thread.java:750)
- java.lang.NumberFormatException: multiple points
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1914)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at java.lang.Float.valueOf(Float.java:416)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:131)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
at java.lang.Thread.run(Thread.java:750)
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40)
at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39)
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99)
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119)
at com.alibaba.datax.core.Engine.start(Engine.java:92)
at com.alibaba.datax.core.Engine.entry(Engine.java:171)
at com.alibaba.datax.core.Engine.main(Engine.java:204)
Caused by: java.lang.NumberFormatException: multiple points
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1914)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at java.lang.Float.valueOf(Float.java:416)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask.init(ConcurrentTableWriterTask.java:131)
at com.alibaba.datax.plugin.writer.oceanbasev10writer.OceanBaseV10Writer$Task.init(OceanBaseV10Writer.java:210)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
at java.lang.Thread.run(Thread.java:750)