observe 进程core,分析根原

【 使用环境 】生产环境
【 OB or 其他组件 】observer
【 使用版本 】OceanBase_CE 4.2.1.10 (r110000072024112010-28c1343085627e79a4f13c29121646bb889cf901) (Built Nov 20 2024 10:14:47)
【问题描述】observer 进程宕,生成core文件
【复现路径】无法复现




CRASH堆栈
addr2line -pCfe /home/admin/oceanbase/bin/observer 0x10bd0390 0xffffad0909ac 0x10e33878 0xa990f38 0xa9900e0 0xa98fffc 0xf69f830 0xf68e35c 0xf68e008 0xada5aac 0xada5540 0xaa3d3d0 0xaa3b700 0xaa37790 0xaa3457c 0x5bd1e28 0x8f7101c 0x8f702a8 0xbf88f64 0xbf87170 0x8825880 0x8820a3c 0xa9a11e8 0xa98b5b4 0x84b4a7c 0x10e6c98
oceanbase::common::light_backtrace(void**, int, long) at easy_util.c:?
?? ??:0
ob_date_unit_type_str at easy_util.c:?
oceanbase::lib::MemoryContext::deinit() at easy_util.c:?
oceanbase::lib::MemoryContext::destory_context(oceanbase::lib::MemoryContext) at easy_util.c:?
oceanbase::lib::_S<(oceanbase::lib::ContextSource)1>::~_S() at easy_util.c:?
oceanbase::common::ObKVCacheStore::try_flush_washable_mb(unsigned long, oceanbase::lib::ObICacheWasher::ObCacheMemBlock
&, long, long, bool) at easy_util.c:?
oceanbase::common::ObKVCacheStore::sync_wash_mbs(unsigned long, long, oceanbase::lib::ObICacheWasher::ObCacheMemBlock*&) at easy_util.c:?
oceanbase::common::ObKVGlobalCache::sync_wash_mbs(unsigned long, long, oceanbase::lib::ObICacheWasher::ObCacheMemBlock*&) at easy_util.c:?
oceanbase::lib::ObTenantMemoryMgr::alloc_chunk(long, oceanbase::lib::ObMemAttr const&) at easy_util.c:?
oceanbase::lib::ObTenantCtxAllocator::alloc_chunk(long, oceanbase::lib::ObMemAttr const&) at easy_util.c:?
oceanbase::lib::BlockSet::alloc_chunk(unsigned long, oceanbase::lib::ObMemAttr const&) at easy_util.c:?
oceanbase::lib::ObjectSet::alloc_block(unsigned long, oceanbase::lib::ObMemAttr const&) at easy_util.c:?
void* oceanbase::lib::ObTenantCtxAllocator::common_allococeanbase::lib::ObjectSet(long, oceanbase::lib::ObMemAttr const&, oceanbase::lib::ObTenantCtxAllocator&, oceanbase::lib::ObjectSet&) at easy_util.c:?
oceanbase::common::ObAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at easy_util.c:?
oceanbase::common::ObParallelAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at easy_util.c:?
oceanbase::sql::ObPsCache::add_stmt_info(oceanbase::sql::ObPsStmtItem const&, oceanbase::sql::ObPsStmtInfo const&, oceanbase::sql::ObPsStmtInfo*&) at easy_util.c:?
oceanbase::sql::ObPsCache::get_or_add_stmt_info(oceanbase::sql::PsCacheInfoCtx const&, oceanbase::sql::ObResultSet const&, oceanbase::share::schema::ObSchemaGetterGuard&, oceanbase::sql::ObPsStmtItem*, oceanbase::sql::ObPsStmtInfo*&) at easy_util.c:?
oceanbase::sql::ObSql::handle_ps_prepare(oceanbase::common::ObString const&, oceanbase::sql::ObSqlCtx&, oceanbase::sql::ObResultSet&, bool) at easy_util.c:?
oceanbase::sql::ObSql::stmt_prepare(oceanbase::common::ObString const&, oceanbase::sql::ObSqlCtx&, oceanbase::sql::ObResultSet&, bool) at easy_util.c:?
oceanbase::observer::ObMPStmtPrepare::do_process(oceanbase::sql::ObSQLSessionInfo&, bool, bool, bool&) at easy_util.c:?
oceanbase::observer::ObMPStmtPrepare::process() at easy_util.c:?
oceanbase::omt::ObWorkerProcessor::process(oceanbase::rpc::ObRequest&) at easy_util.c:?
oceanbase::omt::ObThWorker::worker(long&, long&, int&) at easy_util.c:?
non-virtual thunk to oceanbase::omt::ObThWorker::run(long) at easy_util.c:?
?? ??:0

2 个赞

把当时的observer.log日志 也发一下吧 能把core dump打包发一下么?安装一下debug-info包用gdb分析一下 core dump文件
https://mirrors.oceanbase.com/community/stable/

按需根据操作系统的类型和 oceanbase的版本下载对应的 oceanbase-ce-debuginfo 包

3 个赞

含有crash的observer日志 observer.log.7z (12.4 MB)

3 个赞

老师, 操作系统: UnionTech OS Server 20 aarch64 架构 observe版本: OceanBase_CE 4.2.1.10 (r110000072024112010-28c1343085627e79a4f13c29121646bb889cf901) (Built Nov 20 2024 10:14:47) 应该下载哪个debug-info

3 个赞

根据这个地址 下载对应的aarch64 架构的版本

gdb解析core文件结果
gdb-out.tar.gz (210.9 KB)

看了一下 应该是个已知的问题 从代码侧看的话 应该是在435bp2修复了 目前421.x的版本 目前社区不在维护了 建议升级高版本 建议使用425.x和435.x或者442.x的LTS版本

老师,具体问题是什么呢,有没有临时的规避方案

触发 SQL:约 50KB 的多值 INSERT INTO sjnldz_cloud.dm_d_statemark_cb_pop_user_info_part ...,sql_id=E8326A4D26E09CF5D84896948FFFCA1A导致的内存不足进而crash error了

临时的方案:

  • 拆小 batch INSERT / 控制 PS 大语句;扩大租户 1036 内存