OMS store内存一直增加,直到OOM,无法恢复

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】
omS
oms_4.2.3
ob->mysql
ob 4.2.x版本

store的内存在使用过程中内存占用一直增加,一直到机器OOM,机器内存是128G
刚起来没多久就占用75G
40659 1000 20 0 94.843g 0.075t 27800 S 183.9 61.8 1888:11 store

1 个赞

40659 1000 20 0 96.211g 0.077t 27824 S 168.5 62.8 1921:36 store

5分钟后涨到77G了,TOP命令

看看,当前,运行中的任务情况

怎么看运行情况?目前都是运行中,

|10.88.24.178-7100:OB_MYSQL_CE_np_5sphlh4zt3b4_5sphlwo5kwgw-1-0:0000000007|OceanBase-CE|15,981|248,767|运行中|暂停

更新

查看日志
起始位点:2024-06-07 15:59:59

当前位点:2024-06-08 14:07:48

连接数:0|

现在已经80G了,没多久就会OOM了
40659 1000 20 0 99.617g 0.080t 27776 S 885.1 65.6 2028:27 store

40659 1000 20 0 0.103t 0.085t 27788 S 222.2 69.5 2087:42 store
这会85G了,这有没有办法解决、

全量还是增量同步。
截图,看看呗

store是在拉增量数据,截图有水印,刚才已经发过store状态了

store查看组件,更新,配置参数发一下?

libobcdc.log可以发下吗?

“drc_frame.dbversion”:

string"4.2.1.3"

“drc_frame.log4cpp_category”:

string"log"

“drc_frame.log4cpp_cfg”:

string"./conf/logger.conf"

“drc_frame.modules_name”:

string"ob2store"

“drc_frame.modules_path”:

string"./lib64/reader/ob-ce-4.x-reader"

“drc_frame.monitor_port”:

string"0"

“global.config.version”:

string"3"

“global.running.mode”:

string"strict"

“liboblog.cluster_appname”:

string"fcpObCommonCluster"

“liboblog.cluster_db_name”:

string"oceanbase"

“liboblog.cluster_password”:

string"******"

“liboblog.cluster_url”:

string"http://10.14.10.241:8191/services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=fcpObCommonCluster"

“liboblog.cluster_user”:

string"root"

“liboblog.enable_output_hidden_primary_key”:

string"1"

“liboblog.enable_output_invisible_column”:

string"1"

“liboblog.enable_output_trans_order_by_sql_operation”:

string"1"

“liboblog.first_start_timestamp”:

string"1717661176000"

“liboblog.history_schema_version_count”:

string"16"

“liboblog.instance_index”:

string"0"

“liboblog.instance_num”:

string"1"

“liboblog.progress_limit_sec_for_ddl”:

string"3600"

“liboblog.region”:

string"默认地域"

“liboblog.sort_trans_participants”:

string"1"

“liboblog.target_ob_region”:

string"default"

“liboblog.tb_black_list”:

string"..delay_delete_|..DELAY_DELETE_"

“liboblog.tb_white_list”:

string"merchantMarketing.gifshow.order_quote_bill_mapping|merchantMarketing.gifshow.coupon_receive_buyer|merchantMarketing.gifshow.marketing_budget_refund_flow|merchantMarketing.gifshow.marketing_card_voucher|merchantMarketing.gifshow.mr_acquire|merchantMarketing.gifshow.marketing_budget_use_flow|merchantMarketing.gifshow.cash_assets_trade_refund_record|merchantMarketing.gifshow.order_quote_bill|merchantMarketing.gifshow.promotion_charge|merchantMarketing.gifshow.cash_assets_trade_pay_record|merchantMarketing.gifshow.marketing_card_voucher_flow|merchantMarketing.gifshow.marketing_report_entry|merchantMarketing.gifshow.promotion_activity|merchantMarketing.gifshow.promotion_user"

“liboblog.timezone”:

string"+08:00"

“ob2store.collect_ddl”:

string"true"

“ob2store.dbtype”:

string"oceanbase"

“ob2store.error.level”:

string"WARN"

“ob2store.master.binlog”:

string"1717749923"

“ob2store.master.host”:

string"1.1.1.1"

“ob2store.master.offset”:

string"170215"

“ob2store.master.port”:

string"1"

“ob2store.master.timestamp”:

string"1717749928"

“ob2store.parallelism”:

string"1024"

“ob2store.pipeline”:

string"reset,read,parse,filter|consume"

“ob2store.serialize_pool_size”:

string"4"

“ob2store.subId”:

string"0000000007"

“ob2store.subTopic”:

string"OB_MYSQL_CE_np_5sphlh4zt3b4_5sphlwo5kwgw-1-0"

“ob2store.topic”:

string"OB_MYSQL_CE_np_5sphlh4zt3b4_5sphlwo5kwgw"

“store.clearer.outdated”:

string"432000"

“store.clearer.period”:

string"3600"

“store.client.wait”:

string"43200"

“store.connection.numLimit”:

string"1000"

“store.drcnet.threadPoolSize”:

string"2"

“store.drcnetListenPort”:

string"17003"

“store.listeningPort”:

string"17002"

“store.queue.forceIndexIter”:

string"1"

“store.queue.threadPoolSize”:

string"4"

“store.reader”:

string"on"

“store.repStatus”:

string"master"

“store.useThreadPool”:

string"true"

“store.writer.threshold”:

string"1"

“store.writer.type”:

string"message"

}

libobcdc.log.tar.gz (4.2 MB)

40659 1000 20 0 0.108t 0.091t 27736 S 352.0 74.8 2378:03 store
现在91G了,没多久就会OOM

现在已经OOM了,这个还没法恢复
这种问题有人解决吗?

正在看的

这个问题经过分析确认是store这里的一个内存泄露问题导致的,月底发布的OMS 4.2.4 CE会修复这个问题

现在要用的话?这些任务怎么办?

这问题就是扔在这没人管了?

换个版本吧,用4.2.0的,这个部署很快

就现在跑着的任务需要全部重新部署是吗?