测试环境，项目是obd黑屏安装的，只部署了一个observe，创建了租户。然后采用myloader形式导入数据，导入数据400G左右，昨天 还是正常的可用，今天发现muloader进程还在，但是客户端用租户形式登录登不了，卡着不动。如果用系统租户则可以登录。kill掉了myloader，obd cluster restart了后也是一样租户登着不动，系统租户则可以。这种该如何处理？
dbeaver使用租户去连接就是超时，服务器无返回包。直接用obclient连接就像下面一样无反应，但是可以使用系统用户连接查询进程是可以查到的
可能是队列积压，observer版本是什么，是直连方式连接的么，需要看下日志
可以先在observer的log目录下用如下代码看下是否有积压，${tenantid}替换成业务租户id
grep ' dump tenant info(tenant={id:${tenantid}' observer.log
是tenantid哦不是名称，tenantid可以使用sys租户通过select * from oceanbase.__all_tenant;查询
看起来是存在队列解压，现在外部的workload是否都停了
外部的workload 是指导数码？导数的进程已经停掉了
可以先用pstack（GitHub - peadar/pstack: Print stack traces from running processes, or core files. Works properly for x86_64 without frame pointers ）抓下堆栈看下是阻塞在哪了，这个工具不可以在生产环境使用，切记
pstack $(pgrep observer) > /tmp/pstack.log
这个查完的后半部分信息，看不懂 
，老师帮忙瞧瞧
#0 0x00002b289320cc89 in syscall () from /lib64/libc.so.6
#1 0x00000000039d0aed in futex_hook ()
#2 0x0000000003adb0db in oceanbase::common::ObPriorityQueue2<0, 1, 0>::do_pop(oceanbase::common::ObLink*&, long, long) ()
#3 0x00000000039e1d1a in oceanbase::omt::ObThWorker::worker(long&, long&, int&) ()
#4 0x0000000006339f84 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) ()
#5 0x000000000dc6a1a7 in std::_Function_handler<void (), oceanbase::lib::Threads::start()::$_156>::_M_invoke(std::_Any_data const&) ()
#6 0x000000000dc6402a in oceanbase::lib:
:__th_start(void*) ()
#7 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x2b2c5187b300 (LWP 24267)):
#0 0x00002b289320cc89 in syscall () from /lib64/libc.so.6
#1 0x00000000039d0aed in futex_hook ()
#2 0x0000000003adb0db in oceanbase::common::ObPriorityQueue2<0, 1, 0>::do_pop(oceanbase::common::ObLink*&, long, long) ()
#3 0x00000000039e1d1a in oceanbase::omt::ObThWorker::worker(long&, long&, int&) ()
#4 0x0000000006339f84 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) ()
#5 0x000000000dc6a1a7 in std::_Function_handler<void (), oceanbase::lib::Threads::start()::$_156>::_M_invoke(std::_Any_data const&) ()
#6 0x000000000dc6402a in oceanbase::lib::__th_start(void*) ()
#7 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x2b2c518f7300 (LWP 24268)):
#0 0x00002b289320cc89 in syscall () from /lib64/libc.so.6
#1 0x00000000039d0aed in futex_hook ()
#2 0x0000000003adb0db in oceanbase::common::ObPriorityQueue2<0, 1, 0>::do_pop(oceanbase::common::ObLink*&, long, long) ()
#3 0x00000000039e1d1a in oceanbase::omt::ObThWorker::worker(long&, long&, int&) ()
#4 0x0000000006339f84 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) ()
#5 0x000000000dc6a1a7 in std::_Function_handler<void (), oceanbase::lib::Threads::start()::$_156>::_M_invoke(std::_Any_data const&) ()
#6 0x000000000dc6402a in oceanbase::lib::__th_start(void*) ()
#7 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x2b2c51973300 (LWP 24269)):
#0 0x00002b289320cc89 in syscall () from /lib64/libc.so.6
#1 0x00000000039d0aed in futex_hook ()
#2 0x0000000003adb0db in oceanbase::common::ObPriorityQueue2<0, 1, 0>::do_pop(oceanbase::common::ObLink*&, long, long) ()
#3 0x00000000039e1d1a in oceanbase::omt::ObThWorker::worker(long&, long&, int&) ()
#4 0x0000000006339f84 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) ()
#5 0x000000000dc6a1a7 in std::_Function_handler<void (), oceanbase::lib::Threads::start()::$_156>::_M_invoke(std::_Any_data const&) ()
#6 0x000000000dc6402a in oceanbase::lib::__th_start(void*) ()
#7 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x2b2902d73300 (LWP 24313)):
#0 0x00002b2892f03de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000003be80c7 in bool obutil::Cond::timed_wait_implobutil::ObUtilMutex(obutil::ObUtilMutex const&, obutil::ObSysTime const&) const ()
#2 0x0000000003bfed76 in obutil::ObMonitorobutil::ObUtilMutex::timed_wait(obutil::ObSysTime const&) const ()
#3 0x0000000003bf9708 in oceanbase::common::ObTimer::run1() ()
#4 0x000000000dc671e7 in oceanbase::lib::Threads::run(long) ()
#5 0x000000000dc6402a in oceanbase::lib::__th_start(void*) ()
#6 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#7 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x2b29073ef300 (LWP 24314)):
#0 0x00002b2892f03de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000dc67679 in ob_pthread_cond_timedwait ()
#2 0x0000000003cebbd6 in oceanbase::common::ObThreadCond::wait_us(unsigned long) ()
#3 0x0000000006527e73 in oceanbase::rootserver::ObThreadIdling::idle(long) ()
#4 0x0000000006d9c6bb in oceanbase::rootserver::ObDDLScheduler::run1() ()
#5 0x000000000dc671e7 in oceanbase::lib::Threads::run(long) ()
#6 0x000000000dc6402a in oceanbase::lib::__th_start(void*) ()
#7 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x2b2908e7b300 (LWP 24315)):
#0 0x00002b2892f03de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000dc67679 in ob_pthread_cond_timedwait ()
#2 0x0000000003cebbd6 in oceanbase::common::ObThreadCond::wait_us(unsigned long) ()
#3 0x0000000006527e73 in oceanbase::rootserver::ObThreadIdling::idle(long) ()
#4 0x0000000006d9c6bb in oceanbase::rootserver::ObDDLScheduler::run1() ()
#5 0x000000000dc671e7 in oceanbase::lib::Threads::run(long) ()
#6 0x000000000dc6402a in oceanbase::lib::th_start(void*) ()
#7 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x2b2908ef7300 (LWP 24316)):
#0 0x00002b28931d985d in nanosleep () from /lib64/libc.so.6
#1 0x00002b289320a134 in usleep () from /lib64/libc.so.6
#2 0x0000000003a6f4e7 in oceanbase::transaction::ObTransService::sync_acquire_global_snapshot(oceanbase::transaction::ObTxDesc&, long, oceanbase::s
hare::SCN&, long&) ()
#3 0x0000000003a6d5c9 in oceanbase::transaction::ObTransService::get_read_snapshot(oceanbase::transaction::ObTxDesc&, oceanbase::transaction::ObTxIs
olationLevel, long, oceanbase::transaction::ObTxReadSnapshot&) ()
#4 0x000000000a040738 in oceanbase::sql::ObSqlTransControl::stmt_setup_snapshot(oceanbase::sql::ObSQLSessionInfo*, oceanbase::sql::ObDASCtx&, oceanbase::sql::ObPhysicalPlan const*, oceanbase::sql::ObPhysicalPlanCtx const*, oceanbase::transaction::ObTransService*) ()
#5 0x0000000003a6c86f in oceanbase::sql::ObSqlTransControl::start_stmt(oceanbase::sql::ObExecContext&) ()
#6 0x0000000003a678a1 in oceanbase::sql::ObResultSet::execute() ()
#7 0x0000000003a3be50 in oceanbase::sql::ObResultSet::open() ()
#8 0x0000000003d00123 in oceanbase::observer::ObInnerSQLResult::open() ()
#9 0x0000000003cff573 in oceanbase::observer::ObInnerSQLConnection::do_query(oceanbase::common::sqlclient::ObIExecutor&, oceanbase::observer::ObInne
rSQLResult&) ()
#10 0x0000000003cfe08c in oceanbase::observer::ObInnerSQLConnection::query(oceanbase::common::sqlclient::ObIExecutor&, oceanbase::observer::ObInnerSQ
LResult&, oceanbase::observer::ObVirtualTableIteratorFactory*) ()
#11 0x0000000005c447d8 in oceanbase::observer::ObInnerSQLConnection::execute_read_inner(long, unsigned long, oceanbase::common::ObString const&, ocea
nbase::common::ObISQLClient::ReadResult&, bool, oceanbase::common::ObAddr const*) ()
#12 0x0000000003cfbe8b in oceanbase::observer::ObInnerSQLConnection::execute_read(long, unsigned long, oceanbase::common::ObString const&, oceanbase:
:common::ObISQLClient::ReadResult&, bool, oceanbase::common::ObAddr const*) ()
#13 0x0000000003cfbca2 in oceanbase::observer::ObInnerSQLConnection::execute_read(unsigned long, char const*, oceanbase::common::ObISQLClient::ReadRe
sult&, bool, oceanbase::common::ObAddr const*) ()
#14 0x0000000003cfba2e in oceanbase::common::ObCommonSqlProxy::read(oceanbase::common::sqlclient::ObISQLConnection*, oceanbase::common::ObISQLClient:
:ReadResult&, unsigned long, char const*, oceanbase::common::ObAddr const*) ()
#15 0x0000000003cfb890 in oceanbase::common::ObCommonSqlProxy::read(oceanbase::common::ObISQLClient::ReadResult&, unsigned long, char const*) ()
#16 0x0000000005ca9cd4 in oceanbase::common::ObSQLClientRetryWeak::read(oceanbase::common::ObISQLClient::ReadResult&, unsigned long, char const*) ()
#17 0x0000000007d05ae9 in oceanbase::observer::ObIterateVirtualTable::next_tenant() ()
#18 0x0000000007d06e69 in oceanbase::observer::ObIterateVirtualTable::inner_get_next_row(oceanbase::common::ObNewRow*&) ()
#19 0x0000000003b86095 in oceanbase::common::ObVirtualTableIterator::get_next_row(oceanbase::common::ObNewRow*&) ()
#20 0x0000000003b84b00 in oceanbase::common::ObVirtualTableIterator::get_next_row() ()
#21 0x0000000003b83c12 in oceanbase::sql::ObTableScanOp::inner_get_next_row_for_tsc() ()
#22 0x0000000003b83882 in oceanbase::sql::ObTableScanOp::inner_get_next_row_implement() ()
#23 0x0000000003d344d1 in oceanbase::sql::ObTableScanOp::inner_get_next_row() ()
#24 0x0000000003b65065 in oceanbase::sql::ObOperator::get_next_row() ()
#25 0x00000000039f8966 in oceanbase::sql::ObExecuteResult::get_next_row(oceanbase::sql::ObExecContext&, oceanbase::common::ObNewRow const*&) ()
#26 0x0000000003d0059b in oceanbase::sql::ObResultSet::get_next_row(oceanbase::common::ObNewRow const*&) ()
#27 0x0000000003d0015d in oceanbase::observer::ObInnerSQLResult::open() ()
#28 0x0000000003cff573 in oceanbase::observer::ObInnerSQLConnection::do_query(oceanbase::common::sqlclient::ObIExecutor&, oceanbase::observer::ObInne
rSQLResult&) ()
#29 0x0000000003cfe08c in oceanbase::observer::ObInnerSQLConnection::query(oceanbase::common::sqlclient::ObIExecutor&, oceanbase::observer::ObInnerSQ
LResult&, oceanbase::observer::ObVirtualTableIteratorFactory*) ()
#30 0x0000000005c447d8 in oceanbase::observer::ObInnerSQLConnection::execute_read_inner(long, unsigned long, oceanbase::common::ObString const&, ocea
nbase::common::ObISQLClient::ReadResult&, bool, oceanbase::common::ObAddr const*) ()
#31 0x0000000003cfbe8b in oceanbase::observer::ObInnerSQLConnection::execute_read(long, unsigned long, oceanbase::common::ObString const&, oceanbase:
:common::ObISQLClient::ReadResult&, bool, oceanbase::common::ObAddr const*) ()
#32 0x0000000003cfbca2 in oceanbase::observer::ObInnerSQLConnection::execute_read(unsigned long, char const*, oceanbase::common::ObISQLClient::ReadResult&, bool, oceanbase::common::ObAddr const*) ()
#33 0x0000000003cfba2e in oceanbase::common::ObCommonSqlProxy::read(oceanbase::common::sqlclient::ObISQLConnection*, oceanbase::common::ObISQLClient::ReadResult&, unsigned long, char const*, oceanbase::common::ObAddr const*) ()
#34 0x0000000003cfb890 in oceanbase::common::ObCommonSqlProxy::read(oceanbase::common::ObISQLClient::ReadResult&, unsigned long, char const*) ()
#35 0x0000000006dcb7be in oceanbase::rootserver::ObDDLTaskRecordOperator::get_task_record(oceanbase::common::ObSqlString const&, oceanbase::common::ObMySQLProxy&, oceanbase::common::ObIAllocator&, oceanbase::common::ObIArrayoceanbase::rootserver::ObDDLTaskRecord&) ()
#36 0x0000000006d97e06 in oceanbase::rootserver::ObDDLScheduler::recover_task() ()
#37 0x0000000006d97b71 in oceanbase::rootserver::ObDDLScheduler::DDLScanTask::runTimerTask() ()
#38 0x0000000003bf8fa5 in oceanbase::common::ObTimer::run1() ()
#39 0x000000000dc671e7 in oceanbase::lib::Threads::run(long) ()
#40 0x000000000dc6402a in oceanbase::lib::__th_start(void*) ()
#41 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#42 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x2b2908f73300 (LWP 24317)):
#0 0x00002b2892f03de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000003be80c7 in bool obutil::Cond::timed_wait_implobutil::ObUtilMutex(obutil::ObUtilMutex const&, obutil::ObSysTime const&) const ()
#2 0x0000000003bfed76 in obutil::ObMonitorobutil::ObUtilMutex::timed_wait(obutil::ObSysTime const&) const ()
#3 0x0000000003bf9708 in oceanbase::common::ObTimer::run1() ()
#4 0x000000000dc671e7 in oceanbase::lib::Threads::run(long) ()
#5 0x000000000dc6402a in oceanbase::lib::__th_start(void*) ()
#6 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#7 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x2b2908fef300 (LWP 24319)):
#0 0x00002b2892f03de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000dc67679 in ob_pthread_cond_timedwait ()
#2 0x0000000003cebbd6 in oceanbase::common::ObThreadCond::wait_us(unsigned long) ()
#3 0x00000000069a07ad in oceanbase::observer::ObUniqTaskQueue<oceanbase::rootserver::ObDRTaskTableUpdateTask, oceanbase::rootserver::ObDRTaskTableUpdater>::run1() ()
#4 0x000000000dc671e7 in oceanbase::lib::Threads::run(long) ()
#5 0x000000000dc6402a in oceanbase::lib::__th_start(void*) ()
#6 0x00002b2892effea5 in start_thread () from /lib64/libpthread.so.0
#7 0x00002b289321296d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b2956173300 (LWP 81562)):
#0 0x00002b28931d985d in nanosleep () from /lib64/libc.so.6
#1 0x00002b28931d96f4 in sleep () from /lib64/libc.so.6
#2 0x0000000007d7be89 in oceanbase::observer::ObServer::wait() ()
#3 0x000000000598bc3d in main ()
这段堆栈里没有找到工作进程，你看下现在是否还是有积压，就是queue[x]=后面的数字是否为0