设备单机部署，observer启动失败

OceanBase
#1

【产品名称】

observer

【产品版本】

3.1.2

【问题描述】

obd cluster start XXX

obsever启动后进程死掉，observer.log日志信息如下：

[2022-3-15 8:52:48.818331] INFO [run1@ob_signal_worker.cpp:203] [293248][Y7369676E616C-1] receive request, req=1, accumulated=1

[2022-3-15 8:52:48.860759] WARN [wait_readable@ob_signal_utils.cpp:269] [289909][Y7369676E616C-5] poll failed, errno=4

[2022-3-15 8:52:48.860774] WARN [send_request_and_wait@ob_signal_worker.cpp:78] [289909][Y7369676E616C-5] wait_readable failed, ret=-4015

[2022-3-15 8:52:48.860899] WARN [wait_readable@ob_signal_utils.cpp:269] [289910][Y7369676E616C-3] poll failed, errno=4

[2022-3-15 8:52:48.860916] WARN [send_request_and_wait@ob_signal_worker.cpp:78] [289910][Y7369676E616C-3] wait_readable failed, ret=-4015

[2022-3-15 8:52:48.861752] WARN [wait_readable@ob_signal_utils.cpp:269] [289919][Y7369676E616C-2] poll failed, errno=4

[2022-3-15 8:52:48.861767] WARN [send_request_and_wait@ob_signal_worker.cpp:78] [289919][Y7369676E616C-2] wait_readable failed, ret=-4015

[2022-3-15 8:52:48.862676] WARN [wait_readable@ob_signal_utils.cpp:269] [289929][Y7369676E616C-4] poll failed, errno=4

[2022-3-15 8:52:48.862687] WARN [send_request_and_wait@ob_signal_worker.cpp:78] [289929][Y7369676E616C-4] wait_readable failed, ret=-4015

[2022-3-15 8:52:49.379753] INFO [run1@ob_signal_worker.cpp:203] [293248][Y7369676E616C-2] receive request, req=1, accumulated=2

CRASH ERROR!!! sig=11, sig_code=128, sig_addr=0, timestamp=1647334369379768, tid=289922, tname=DAG_MINI_MERGE, trace_id=16675193749505-1647334423919443, extra_info=((null)), lbt=0x99b1188 0x99b1e68 0x7f0aaadaec1f 0x9c218ef 0x9c0f972 0x9bac07d 0x9bb585c 0x9b653da 0x957f6b3 0x95acb5a 0x81e4d25 0x82200ec 0x821e45e 0x821e045 0x87330d8 0x8740cab 0x7b9b095 0x33956f9 0x3398b6f 0x336e3fe 0x2c5b901 0x970da04 0x970c731 0x970924e

#2

麻烦提供一下：

1、observer的日志打包后上传一下附件。

2、obd使用的yaml附件。

3、df -h && free -g && grep processor /proc/cpuinfo | wc -l

#3

observer log


log.zip (2790324 KB)

#4

yaml

oceanbase-ce:

 servers:

  • 127.0.0.1

 global:

  home_path: /nvme1n1/ob_data_qpl_16K

  devname: lo

  mysql_port: 3881

  rpc_port: 3882

  zone: zone1

  cluster_id: 1

  memory_limit: 110G



[root@spr46 load]# df -h && free -g && grep processor /proc/cpuinfo | wc -l

Filesystem   Size Used Avail Use% Mounted on

devtmpfs    126G   0 126G  0% /dev

tmpfs      126G   0 126G  0% /dev/shm

tmpfs      126G  29M 126G  1% /run

tmpfs      126G   0 126G  0% /sys/fs/cgroup

/dev/sda4    889G 308G 581G 35% /

/dev/sda2   1014M 332M 683M 33% /boot

/dev/sda1    599M 7.3M 592M  2% /boot/efi

tmpfs      26G  16K  26G  1% /run/user/42

tmpfs      26G   0  26G  0% /run/user/0

/dev/nvme0n1  1.8T 1.4T 370G 79% /nvme0n1

/dev/nvme1n1  1.8T 245G 1.5T 15% /nvme1n1

total    used    free   shared buff/cache  available

Mem:      251     10     240      0      1     239

Swap:       3      0      3

224


#5

observer.log日志不完整，有没有完整的。

#6

 

log.zip (1318553 KB)

#7

addr2line -Cfe ./observer 0x99b1188 0x99b1e68 0x7f0b6e041c1f 0x9c218ef 0x9c0f972 0x9bac07d 0x9bb585c 0x9b653da 0x957f6b3 0x95acb5a 0x81e4d25 0x82200ec 0x821e45e 0x821e045 0x87330d8 0x8740cab 0x7b9b095 0x33956f9 0x3398b6f 0x336e3fe 0x2c5b901 0x970da04 0x970c731 0x970924e



将这个栈的信息发一下。

#8

Curl_http

http.c:?

Curl_hash_clean

hash.c:?

SXNET_add_id_asc

??:?

subre(vars*, int, int, state*, state*)

./build_release/deps/oblib/src/lib/regex/./deps/oblib/src/lib/regex/regex/regcomp.cpp:1653

oceanbase::common::ObSeqQueue::init(long, oceanbase::common::ObSeqQueue::SeqItem*)

./build_release/deps/oblib/src/lib/./deps/oblib/src/lib/queue/ob_seq_queue.cpp:37

oceanbase::blocksstable::ObIMicroBlockCache::ObIMicroBlockIOCallback::put_cache_and_fetch(oceanbase::blocksstable::ObFullMacroBlockMeta const&, oceanbase::blocksstable::ObMacroBlockReader&, char const*, long, long, char const*, long, oceanbase::blocksstable::ObMicroBlockCacheValue const*&, oceanbase::common::ObKVCacheHandle&)

./build_release/src/storage/./deps/oblib/src/lib/oblog/ob_log.h:1123

int oceanbase::common::logdata_print_key_obj<oceanbase::blocksstable::MacroBlockId>(char*, long, long&, char const*, bool, oceanbase::blocksstable::MacroBlockId const&, oceanbase::common::BoolType<false>)

./build_release/src/storage/./deps/oblib/src/lib/oblog/ob_log_print_kv.h:331

void oceanbase::common::OB_PRINT<int, oceanbase::common::ObLobIndex, oceanbase::blocksstable::ObMacroBlockInfoPair>(char const*, int, char const*, int, char const*, unsigned long, char const*, char const*, int const&, char const*, oceanbase::common::ObLobIndex const&, char const*, oceanbase::blocksstable::ObMacroBlockInfoPair const&)

??:?

void oceanbase::common::OB_PRINT<int, oceanbase::common::ObLobIndex, oceanbase::blocksstable::ObMacroBlockInfoPair>(char const*, int, char const*, int, char const*, unsigned long, char const*, char const*, int const&, char const*, oceanbase::common::ObLobIndex const&, char const*, oceanbase::blocksstable::ObMacroBlockInfoPair const&)

./build_release/src/storage/./deps/oblib/src/lib/oblog/ob_log.h:1463

oceanbase::storage::ObStreamRpcReader<(oceanbase::obrpc::ObRpcPacketCode)1117>::check_need_fetch_next_buffer(bool&)

./build_release/src/storage/./src/storage/ob_partition_base_data_ob_reader.cpp:179

oceanbase::common::ObArrayImpl<oceanbase::blocksstable::ObSSTablePair, oceanbase::common::ModulePageAllocator, false, oceanbase::common::ObArrayDefaultCallBack<oceanbase::blocksstable::ObSSTablePair>, oceanbase::common::DefaultItemEncode<oceanbase::blocksstable::ObSSTablePair> >::assign(oceanbase::common::ObIArray<oceanbase::blocksstable::ObSSTablePair> const&)::{lambda(char const*)#1}::operator()(char const*) const

./build_release/src/storage/./deps/oblib/src/lib/oblog/ob_log.h:1123

oceanbase::storage::ObPartGroupMigrationTask::check_member_major_sstable_enough(oceanbase::common::ObIArray<oceanbase::storage::ObReportPartMigrationTask>&)

./build_release/src/storage/./src/storage/ob_partition_migrator.cpp:4308

oceanbase::share::schema::ObTableIterator::next(unsigned long&)

./build_release/src/share/./src/share/schema/ob_table_iter.cpp:70

oceanbase::share::schema::ColumnMap::init(oceanbase::common::ObIArray<unsigned long> const&)

./build_release/src/share/./src/share/schema/ob_table_param.cpp:191

oceanbase::share::schema::ObServerSchemaService::refresh_tenant_full_normal_schema(oceanbase::common::ObISQLClient&, oceanbase::share::schema::ObRefreshSchemaStatus const&, long)

./build_release/src/share/./src/share/schema/ob_server_schema_service.cpp:5240

oceanbase::share::ObLocationFetcher::renew_location_with_rpc_v2(oceanbase::common::hash::ObHashMap<oceanbase::share::ObReplicaRenewKey, oceanbase::obrpc::ObMemberListAndLeaderArg const*, oceanbase::common::hash::LatchReadWriteDefendMode, oceanbase::common::hash::hash_func<oceanbase::share::ObReplicaRenewKey>, oceanbase::common::hash::equal_to<oceanbase::share::ObReplicaRenewKey>, oceanbase::common::hash::SimpleAllocer<oceanbase::common::hash::ObHashTableNode<oceanbase::common::hash::HashMapPair<oceanbase::share::ObReplicaRenewKey, oceanbase::obrpc::ObMemberListAndLeaderArg const*> >, 77, oceanbase::common::hash::SpinMutexDefendMode, oceanbase::common::hash::DefaultSimpleAllocerAllocator>, oceanbase::common::hash::NormalPointer, oceanbase::common::ObMalloc>*, oceanbase::share::ObPartitionLocation const&, oceanbase::share::ObPartitionLocation&, bool&)

./build_release/src/share/./deps/oblib/src/lib/oblog/ob_log.h:1123

oceanbase::common::hash::ObHashTable<oceanbase::common::ObErasureCodeEncodeKey, oceanbase::common::hash::HashMapPair<oceanbase::common::ObErasureCodeEncodeKey, unsigned char*>, oceanbase::common::hash::hash_func<oceanbase::common::ObErasureCodeEncodeKey>, oceanbase::common::hash::equal_to<oceanbase::common::ObErasureCodeEncodeKey>, oceanbase::common::hash::pair_first<oceanbase::common::hash::HashMapPair<oceanbase::common::ObErasureCodeEncodeKey, unsigned char*> >, oceanbase::common::hash::SimpleAllocer<oceanbase::common::hash::ObHashTableNode<oceanbase::common::hash::HashMapPair<oceanbase::common::ObErasureCodeEncodeKey, unsigned char*> >, 100, oceanbase::common::hash::SpinMutexDefendMode, oceanbase::common::hash::DefaultSimpleAllocerAllocator>, oceanbase::common::hash::LatchReadWriteDefendMode, oceanbase::common::hash::NormalPointer, oceanbase::common::ObMalloc, 1l>::internal_get_with_timeout_unsafe(oceanbase::common::hash::ObHashTableBucket<oceanbase::common::hash::HashMapPair<oceanbase::common::ObErasureCodeEncodeKey, unsigned char*>, oceanbase::common::ObLatch, oceanbase::common::hash::NCond>&, oceanbase::common::ObErasureCodeEncodeKey const&, oceanbase::common::hash::HashMapPair<oceanbase::common::ObErasureCodeEncodeKey, unsigned char*> const*&, long)

./build_release/deps/oblib/src/lib/./deps/oblib/src/lib/hash/ob_hashtable.h:788

operator()

./build_release/deps/oblib/src/lib/./deps/oblib/src/lib/hash/ob_hashtable.h:585

oceanbase::common::ObArrayImpl<oceanbase::common::ObDiskFd, oceanbase::common::ModulePageAllocator, false, oceanbase::common::ObArrayDefaultCallBack<oceanbase::common::ObDiskFd>, oceanbase::common::NotImplementItemEncode<oceanbase::common::ObDiskFd> >::assign(oceanbase::common::ObIArray<oceanbase::common::ObDiskFd> const&)::{lambda(char const*)#1}::operator()(char const*) const

./build_release/deps/oblib/src/lib/./deps/oblib/src/lib/oblog/ob_log.h:1123


#9

感觉有点像内存的原因

另外配置文件写得太简单了，布置个小规格试试，参考https://github.com/oceanbase/obdeploy/blob/master/example/mini-local-example.yaml

1 个赞
#10

好的，谢谢。小规格的配置文件确实可以，但不满足测试需求，我尝试缩减配置规格，看看是否可行。

#11

资源不足导致。