安装 ocp-all-in-one-4.2.1-20231208144448.el7.x86_64 进度条停滞不动

OceanBase 技术问题
#1


【 使用环境 】 测试环境
【 OB or 其他组件 】OCP
【 使用版本 】 ocp-all-in-one-4.2.1-20231208144448.el7
【问题描述】web 安装时一只停滞在 “部署中…”
部署日志的最后部分如下（具体部署画面见附件中的图片）：

±----------------------------------------------+
| obproxy |
±------------±-----±----------------±-------+
| ip | port | prometheus_port | status |
±------------±-----±----------------±-------+
| 10.0.104.30 | 2883 | 2884 | active |
±------------±-----±----------------±-------+
obclient -h10.0.104.30 -P2883 -uroot -p’XA@xa#12’ -Doceanbase -A
succeed
Search plugins ok
Load cluster param plugin ok
Check before start ocp-server ok
Create tenant ocp_meta ok
Create tenant ocp_monitor ok
Start ocp-server ok
ocp-server program health check ok
Start ocp-server ok
ocp-server program health check ok
Connect to ocp-server ok
±-------------------------------------------------------+
| ocp-server |
±------------------------±---------±---------±-------+
| url | username | password | status |
±------------------------±---------±---------±-------+
| http://10.0.104.30:8080 | admin | XA@xa#12 | active |
±------------------------±---------±---------±-------+
succeed

【附件及日志】

image
image1427×1039 81.5 KB

#3

麻烦给下日志呢
还有机器的资源配置。

#4

OCP机器配置: 4核CPU、32G内存、200G硬盘的测试机，完整的日志如下

Package oceanbase-ce-4.2.1.2-102000042023120514.el7 is available.
Package obproxy-ce-4.2.1.0-11.el7 is available.
Package ocp-server-ce-4.2.1-20231208144448.el7 is available.
install oceanbase-ce-4.2.1.2 for local ok
install obproxy-ce-4.2.1.0 for local ok
install ocp-server-ce-4.2.1 for local ok
±--------------------------------------------------------------------------------------------+
| Packages |
±--------------±--------±-----------------------±-----------------------------------------+
| Repository | Version | Release | Md5 |
±--------------±--------±-----------------------±-----------------------------------------+
| oceanbase-ce | 4.2.1.2 | 102000042023120514.el7 | b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 |
| obproxy-ce | 4.2.1.0 | 11.el7 | 0aed4b782120e4248b749f67be3d2cc82cdcb70d |
| ocp-server-ce | 4.2.1 | 20231208144448.el7 | 58cf72891d75a2fa7c754bafc42d336525baf0b5 |
±--------------±--------±-----------------------±-----------------------------------------+
Repository integrity check ok
Parameter check ok
Generate ocp server configuration ok
Cluster status check ok
Initializes observer work home ok
Initializes obproxy work home ok
Initializes ocp-server work home ok
Remote oceanbase-ce-4.2.1.2-102000042023120514.el7-b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 repository install ok
Remote oceanbase-ce-4.2.1.2-102000042023120514.el7-b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 repository lib check !!
Remote obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d repository install ok
Remote obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d repository lib check ok
Remote ocp-server-ce-4.2.1-20231208144448.el7-58cf72891d75a2fa7c754bafc42d336525baf0b5 repository install ok
Remote ocp-server-ce-4.2.1-20231208144448.el7-58cf72891d75a2fa7c754bafc42d336525baf0b5 repository lib check ok
Try to get lib-repository
Package oceanbase-ce-libs-4.2.1.2-102000042023120514.el7 is available.
install oceanbase-ce-libs-4.2.1.2 for local ok
Remote oceanbase-ce-libs-4.2.1.2-102000042023120514.el7-b4ae00ee729404557fa858d4cdd87250bca1aa63 repository install ok
Remote oceanbase-ce-4.2.1.2-102000042023120514.el7-b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 repository lib check ok
myocp deployed
Search plugins ok
Load cluster param plugin ok
Check before start observer ok
[WARN] OBD-1007: (10.0.104.30) The recommended number of stack size is unlimited (Current value: 20480)
[WARN] OBD-1012: (10.0.104.30) clog and data use the same disk (/)

Start observer ok
observer program health check ok
Connect to observer ok
Initialize oceanbase-ce ok
Wait for observer init ok
±----------------------------------------------+
| observer |
±------------±--------±-----±------±-------+
| ip | version | port | zone | status |
±------------±--------±-----±------±-------+
| 10.0.104.30 | 4.2.1.2 | 2881 | zone1 | ACTIVE |
±------------±--------±-----±------±-------+
obclient -h10.0.104.30 -P2881 -uroot -p’XA@xa#12’ -Doceanbase -A

succeed
Search plugins ok
Load cluster param plugin ok
Check before start obproxy ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Initialize obproxy-ce ok
±----------------------------------------------+
| obproxy |
±------------±-----±----------------±-------+
| ip | port | prometheus_port | status |
±------------±-----±----------------±-------+
| 10.0.104.30 | 2883 | 2884 | active |
±------------±-----±----------------±-------+
obclient -h10.0.104.30 -P2883 -uroot -p’XA@xa#12’ -Doceanbase -A
succeed
Search plugins ok
Load cluster param plugin ok
Check before start ocp-server ok
Create tenant ocp_meta ok
Create tenant ocp_monitor ok
Start ocp-server ok
ocp-server program health check ok
Start ocp-server ok
ocp-server program health check ok
Connect to ocp-server ok
±-------------------------------------------------------+
| ocp-server |
±------------------------±---------±---------±-------+
| url | username | password | status |
±------------------------±---------±---------±-------+
| http://10.0.104.30:8080 | admin | XA@xa#12 | active |
±------------------------±---------±---------±-------+
succeed

#5

看提示应该是完成了
obd cluster list 看看。
observer.log 麻烦发下 应该再你部署路径下的oceanbase里log下。

1 个赞
#6

[admin@ob30 setup]$ obd cluster list
±---------------------------------------------------------+
| Cluster List |
±------±-------------------------------±----------------+
| Name | Configuration Path | Status (Cached) |
±------±-------------------------------±----------------+
| myocp | /home/admin/.obd/cluster/myocp | running |
±------±-------------------------------±----------------+
Trace ID: 578b531a-b118-11ee-b2cd-0050568ee2ee
If you want to view detailed obd logs, please run: obd display-trace 578b531a-b118-11ee-b2cd-0050568ee2ee

#7

就是已经部署完成了
可以看下这里的 obd cluster list命令下备注 集群命令组

#8

observer.log 文件具体在那个目录下？我找一下，没有发现有此文件

#9

配置文件：home_path参数指定的路径里
obd cluster edit-config myocp

#10

[admin@ob30 clog]$ obd cluster edit-config myocp
[ERROR] Another app is currently holding the obd lock.
Trace ID: 9cda0942-b119-11ee-9255-0050568ee2ee
If you want to view detailed obd logs, please run: obd display-trace 9cda0942-b119-11ee-9255-0050568ee2ee

[admin@ob30 clog]$ obd display-trace 9cda0942-b119-11ee-9255-0050568ee2ee
[2024-01-12 15:10:05.890] [DEBUG] - cmd: [‘myocp’]
[2024-01-12 15:10:05.890] [DEBUG] - opts: {}
[2024-01-12 15:10:05.890] [DEBUG] - mkdir /home/admin/.obd/lock/
[2024-01-12 15:10:05.890] [DEBUG] - unknown lock mode
[2024-01-12 15:10:05.890] [DEBUG] - try to get share lock /home/admin/.obd/lock/global
[2024-01-12 15:10:05.891] [DEBUG] - share lock /home/admin/.obd/lock/global, count 1
[2024-01-12 15:10:05.891] [DEBUG] - Get Deploy by name
[2024-01-12 15:10:05.891] [DEBUG] - mkdir /home/admin/.obd/cluster/
[2024-01-12 15:10:05.891] [DEBUG] - mkdir /home/admin/.obd/config_parser/
[2024-01-12 15:10:05.891] [DEBUG] - try to get exclusive lock /home/admin/.obd/lock/deploy_myocp
[2024-01-12 15:10:05.892] [ERROR] Another app is currently holding the obd lock.
[2024-01-12 15:10:05.892] [ERROR] Traceback (most recent call last):
[2024-01-12 15:10:05.892] [ERROR] File “_lock.py”, line 64, in _ex_lock
[2024-01-12 15:10:05.892] [ERROR] File “tool.py”, line 498, in exclusive_lock_obj
[2024-01-12 15:10:05.892] [ERROR] BlockingIOError: [Errno 11] Resource temporarily unavailable
[2024-01-12 15:10:05.892] [ERROR]
[2024-01-12 15:10:05.892] [ERROR] During handling of the above exception, another exception occurred:
[2024-01-12 15:10:05.892] [ERROR]
[2024-01-12 15:10:05.892] [ERROR] Traceback (most recent call last):
[2024-01-12 15:10:05.892] [ERROR] File “_lock.py”, line 85, in ex_lock
[2024-01-12 15:10:05.892] [ERROR] File “_lock.py”, line 66, in _ex_lock
[2024-01-12 15:10:05.893] [ERROR] _errno.LockError: [Errno 11] Resource temporarily unavailable
[2024-01-12 15:10:05.893] [ERROR]
[2024-01-12 15:10:05.893] [ERROR] During handling of the above exception, another exception occurred:
[2024-01-12 15:10:05.893] [ERROR]
[2024-01-12 15:10:05.893] [ERROR] Traceback (most recent call last):
[2024-01-12 15:10:05.893] [ERROR] File “obd.py”, line 239, in do_command
[2024-01-12 15:10:05.893] [ERROR] File “obd.py”, line 959, in _do_command
[2024-01-12 15:10:05.893] [ERROR] File “core.py”, line 499, in edit_deploy_config
[2024-01-12 15:10:05.893] [ERROR] File “_deploy.py”, line 1582, in get_deploy_config
[2024-01-12 15:10:05.893] [ERROR] File “_deploy.py”, line 1569, in _lock
[2024-01-12 15:10:05.893] [ERROR] File “_lock.py”, line 283, in deploy_ex_lock
[2024-01-12 15:10:05.893] [ERROR] File “_lock.py”, line 262, in _ex_lock
[2024-01-12 15:10:05.893] [ERROR] File “_lock.py”, line 254, in _lock
[2024-01-12 15:10:05.893] [ERROR] File “_lock.py”, line 185, in lock
[2024-01-12 15:10:05.893] [ERROR] File “_lock.py”, line 90, in ex_lock
[2024-01-12 15:10:05.893] [ERROR] _errno.LockError: [Errno 11] Resource temporarily unavailable
[2024-01-12 15:10:05.893] [ERROR]
[2024-01-12 15:10:05.893] [INFO] Trace ID: 9cda0942-b119-11ee-9255-0050568ee2ee
[2024-01-12 15:10:05.893] [INFO] If you want to view detailed obd logs, please run: obd display-trace 9cda0942-b119-11ee-9255-0050568ee2ee
[2024-01-12 15:10:05.893] [DEBUG] - share lock /home/admin/.obd/lock/global release, count 0
[2024-01-12 15:10:05.893] [DEBUG] - unlock /home/admin/.obd/lock/global
[2024-01-12 15:10:05.893] [DEBUG] - unlock /home/admin/.obd/lock/deploy_myocp

#11

另一个窗口再执行obd命令的原因
/home/admin/.obd/cluster/myocp/config.yaml 看这个吧，根据里面的路径，可以找的到log目录

#12

[admin@ob30 ~]$ cat /home/admin/.obd/cluster/myocp/config.yaml
user:
username: admin
password: ‘123456’
port: 10001
oceanbase-ce:
version: 4.2.1.2
release: 102000042023120514.el7
package_hash: b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4
10.0.104.30:
zone: zone1
servers:

  • 10.0.104.30
    global:
    appname: myocp
    root_password: XA@xa#12
    mysql_port: 2881
    rpc_port: 2882
    home_path: /home/admin/oceanbase
    data_dir: /data/1
    redo_dir: /data/log1
    cluster_id: 1705030052
    proxyro_password: gMgErZmQEs
    enable_syslog_recycle: true
    enable_syslog_wf: false
    max_syslog_file_count: 4
    memory_limit: 23G
    datafile_size: 130G
    system_memory: 5G
    log_disk_size: 59G
    cpu_count: 16
    obproxy-ce:
    version: 4.2.1.0
    package_hash: 0aed4b782120e4248b749f67be3d2cc82cdcb70d
    release: 11.el7
    servers:
  • 10.0.104.30
    global:
    home_path: /admin/obproxy
    prometheus_listen_port: 2884
    listen_port: 2883
    obproxy_sys_password: 95riboqf8y
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    enable_cluster_checkout: false
    depends:
  • oceanbase-ce
    ocp-server-ce:
    version: 4.2.1
    package_hash: 58cf72891d75a2fa7c754bafc42d336525baf0b5
    release: 20231208144448.el7
    servers:
  • 10.0.104.30
    global:
    home_path: /admin/ocp
    soft_dir: /admin/software
    log_dir: /admin/logs
    port: 8080
    admin_password: XA@xa#12
    memory_size: 4G
    manage_info:
    machine: 10
    ocp_meta_tenant:
    tenant_name: ocp_meta
    max_cpu: 2.0
    memory_size: 4G
    ocp_meta_username: root
    ocp_meta_password: XA@xa#12
    ocp_meta_db: meta_database
    ocp_monitor_tenant:
    tenant_name: ocp_monitor
    max_cpu: 2.0
    memory_size: 8G
    ocp_monitor_username: root
    ocp_monitor_password: XA@xa#12
    ocp_monitor_db: monitor_database
    depends:
  • oceanbase-ce
  • obproxy-ce
#13

observer.log 文件的最新一段日志如下

[2024-01-12 15:25:49.961313] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [12630][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26] start to check and clear obj lock(ls->get_ls_id()={id:1})
[2024-01-12 15:25:49.961506] INFO [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1546) [11750][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] execute write sql(ret=0, tenant_id=1, affected_rows=1, sql= update all_weak_read_service set min_version=1705044349760556726, max_version=1705044349760556726 where tenant_id = 1 and level_id = 0 and level_value = ‘’ and min_version = 1705044349560384512 and max_version = 1705044349560384512 )
[2024-01-12 15:25:49.961747] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls (ob_table_lock_service.cpp:213) [12630][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=50] start to check and clear obj lock(ls->get_ls_id()={id:1001})
[2024-01-12 15:25:49.967318] INFO [STORAGE] scheduler_ls_ha_handler (ob_storage_ha_service.cpp:188) [12841][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=33] start do ls ha handler(ls_id_array_=[{id:1}])
[2024-01-12 15:25:49.976617] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:537) [11756][T1_LSArchiveMgr][T1][YB420A00681E-00060EB769A1C8DD-0-0] [lt=26] gc stale ls task succ
[2024-01-12 15:25:49.988263] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [13109][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=47] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1004, min_version={val:1705044349384431557, v:0}, base_version={val:1705044349384431557, v:0}, server_count=1, svr_array=[{addr:“10.0.104.30:2882”, version:{val:1705044349384431558, v:0}, valid_part_count:0, total_part_count:2, generate_tstamp:1705044349988250, is_skipped:true}], skip_server_count=1, skip_servers=[“10.0.104.30:2882”])
[2024-01-12 15:25:49.988292] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:688) [13109][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=28] no need to update min/max version(new_version={val:1705044349384431557, v:0}, min_version={val:1705044349384431557, v:0}, max_version={val:1705044349384431557, v:0})
[2024-01-12 15:25:50.003679] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [12643][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret=“OB_NEED_WAIT”)
[2024-01-12 15:25:50.006331] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [13107][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret=“OB_NEED_WAIT”)
[2024-01-12 15:25:50.013839] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [13102][T1004_LCLSender][T1004][Y0-0000000000000000-0-0] [lt=47] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=106, total_destructed_detector=106, total_alived_detector=0, lcl_op_interval=30000, lcl_msg_map.count()=0, *this={this:0x7fd79c3fa2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0})
[2024-01-12 15:25:50.023156] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [12548][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=25] ====== tenant freeze timer task ======
[2024-01-12 15:25:50.028112] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 50.00)
[2024-01-12 15:25:50.028149] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10801312, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0])
[2024-01-12 15:25:50.028217] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SYS](tenant_id=1002, io_status=sys_group_name: SLOG_IO, mode: write, size: 4096.00, iops: 1.99, rt: 321.50)
[2024-01-12 15:25:50.028233] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CONFIG](tenant_id=1002, ref_cnt=2, io_config={group_num_:0, memory_limit_:3221225472, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=12763552, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0])
[2024-01-12 15:25:50.028296] INFO [COMMON] print_io_status (ob_io_manager.cpp:1393) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SYS](tenant_id=1004, io_status=sys_group_name: SLOG_IO, mode: write, size: 4096.00, iops: 1.99, rt: 274.50)
[2024-01-12 15:25:50.028313] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CONFIG](tenant_id=1004, ref_cnt=2, io_config={group_num_:0, memory_limit_:7516192768, callback_thread_count_:8, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:2}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=12763552, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0])
[2024-01-12 15:25:50.028332] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028344] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028371] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028380] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028387] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028392] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028398] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028404] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028410] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028415] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028421] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028427] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028433] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028438] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028444] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.028450] INFO [COMMON] print_sender_status (ob_io_struct.cpp:846) [11400][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807)
[2024-01-12 15:25:50.040093] INFO [LIB] stat (utility.h:1157) [13083][T1004_ApplySrv0][T1004][YB420A00681E-00060EB778E1FF11-0-0] [lt=9] [APPLY STAT CB APPEND COST TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=19, this=0x7fd76fd5a280, extra_info="")
[2024-01-12 15:25:50.040128] INFO [LIB] stat (utility.h:1157) [13083][T1004_ApplySrv0][T1004][YB420A00681E-00060EB778E1FF11-0-0] [lt=33] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=743, this=0x7fd76fd5a4b0, extra_info="")
[2024-01-12 15:25:50.040148] INFO [LIB] stat (utility.h:1157) [13083][T1004_ApplySrv0][T1004][YB420A00681E-00060EB778E1FF11-0-0] [lt=19] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=0, this=0x7fd76fd5a6e0, extra_info="")
[2024-01-12 15:25:50.040165] INFO [LIB] stat (utility.h:1157) [13083][T1004_ApplySrv0][T1004][YB420A00681E-00060EB778E1FF11-0-0] [lt=16] [APPLY STAT CB EXECUTE TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=-636, this=0x7fd76fd5a910, extra_info="")
[2024-01-12 15:25:50.040181] INFO [LIB] stat (utility.h:1157) [13083][T1004_ApplySrv0][T1004][YB420A00681E-00060EB778E1FF11-0-0] [lt=15] [APPLY STAT CB TOTAL TIME](cur_stat_count=26, stat_interval=5000000, avg (count/cost)=126, this=0x7fd76fd5ab40, extra_info="")
[2024-01-12 15:25:50.054972] INFO [LIB] stat (utility.h:1157) [12662][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7fd7c2221470, extra_info="")
[2024-01-12 15:25:50.055037] INFO [LIB] stat (utility.h:1157) [12662][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT APPEND COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=61, this=0x7fd7da9e6e10, extra_info=“id:1”)
[2024-01-12 15:25:50.061808] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:688) [11750][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] no need to update min/max version(new_version={val:1705044349760556726, v:0}, min_version={val:1705044349760556726, v:0}, max_version={val:1705044349760556726, v:0})
[2024-01-12 15:25:50.064502] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1156) [11424][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-14842634240, global_cache_size=443203584, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004])
[2024-01-12 15:25:50.066849] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=10] ====== checkpoint timer task ======
[2024-01-12 15:25:50.067595] INFO [COMMON] advance_clock (ob_fifo_arena.cpp:374) [12645][T1002_TenantWea][T1001][YB420A00681E-00060EB77162CC42-0-0] [lt=26] current clock is (clock_=0, max_seq_=0, mem_limit=248722220, cur_mem_hold=73400320, attr_.tenant_id_=1001)
[2024-01-12 15:25:50.067664] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [12645][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1705044350067662789, start_id=1705044350067662789, end_id=1705044350067662790)
[2024-01-12 15:25:50.068024] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1389) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=13] succ to get rec scn(*this={this:0x7fd770404030, ls_id:{id:1}, tenant_id:1004, state:“L_WORKING”, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1705034436316038107, v:0}, last_push_gc_task_ts_:1705034483959785, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0})
[2024-01-12 15:25:50.068054] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=29] tx ctx memtable get rec scn(this={ObITable:{this:0x7fd772570080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:“TX_CTX_MEMTABLE”, scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1705034483964830, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7fd772570080, snapshot_version:{val:1705034483964830, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0})
[2024-01-12 15:25:50.068086] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:789) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=26] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1})
[2024-01-12 15:25:50.068176] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:272) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=17] get rec_scn from MdsTableMgr(ret=0, ret=“OB_SUCCESS”, min_rec_scn={val:4611686018427387903, v:0}, min_rec_scn_tablet_ids=[], *this={this:0x7fd772474510, is_inited:true, freezing_scn:{val:1705032641081437411, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1705034488918024754, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1705044347180167829, v:0}, tablet_change_checkpoint_scn:{val:1705030881581434732, v:0}, all_id_meta:{id_meta:[{limited_id:1705044357109793867, latest_log_ts:{val:1705044340865173261, v:0}}, {limited_id:1000001, latest_log_ts:{val:1705030877314138236, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:1, palf_env_:0x7fd7997f2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}})
[2024-01-12 15:25:50.068243] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:507) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=63] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type=“TX_DATA_MEMTABLE_TYPE”, common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7fd7724741b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7fd772479b90, table_type:1, memtable_head:2, memtable_tail:3, t3m:0x7fd8064b2030, tables:[null, null, 0x7fd77252cb90, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:1705032679225490, tx_data_table:0x7fd77247b090, ls_tablet_svr:0x7fd772474190, slice_allocator:0x7fd77247b0d0}, min_rec_scn={val:1705034488918024754, v:0}, ls_id_={id:1})
[2024-01-12 15:25:50.068279] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=28] get rec log scn(service_type_=0, rec_log_ts={val:1705044340865173261, v:0})
[2024-01-12 15:25:50.068290] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0})
[2024-01-12 15:25:50.068306] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:158) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=8] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1705034488918024754, v:0}, checkpoint_scn_in_ls_meta={val:1705034488918024754, v:0}, ls_id={id:1}, service_type=“TRANS_SERVICE”)
[2024-01-12 15:25:50.068327] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1544) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:“10.0.104.30:2882”, has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0})
[2024-01-12 15:25:50.068345] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1})
[2024-01-12 15:25:50.068376] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=20] get_max_applied_scn(scn={val:1705044349885252603, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:36420371}, last_check_scn_:{val:1705044349885252604, v:0}, max_applied_cb_scn_:{val:1705044349885252603, v:0}})
[2024-01-12 15:25:50.068395] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:966) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=16] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1705030881221288951, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1705044349380247, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1705030881221288951, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x7fd76fc8d4f0}, IteratorStorageType::“DiskIteratorStorage”}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}})
[2024-01-12 15:25:50.069546] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1389) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=52] succ to get rec scn(*this={this:0x7fd7704c6030, ls_id:{id:1001}, tenant_id:1004, state:“L_WORKING”, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1705031320585916423, v:0}, last_push_gc_task_ts_:1705032694195782, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0})
[2024-01-12 15:25:50.069572] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=25] tx ctx memtable get rec scn(this={ObITable:{this:0x7fd772570e90, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:“TX_CTX_MEMTABLE”, scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1705032694198973, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7fd772570e90, snapshot_version:{val:1705032694198973, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0})
[2024-01-12 15:25:50.069598] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:789) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=22] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001})
[2024-01-12 15:25:50.069823] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:272) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=15] get rec_scn from MdsTableMgr(ret=0, ret=“OB_SUCCESS”, min_rec_scn={val:4611686018427387903, v:0}, min_rec_scn_tablet_ids=[], *this={this:0x7fd754c5a510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1705031323033072531, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1705044347180167829, v:0}, tablet_change_checkpoint_scn:{val:1705031327192341727, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:1, palf_env_:0x7fd7997f2030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}})
[2024-01-12 15:25:50.069881] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:507) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=55] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type=“DATA_CHECKPOINT_TYPE”, common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x7fd754c68a50}, min_rec_scn={val:1705031323033072531, v:0}, ls_id_={id:1001})
[2024-01-12 15:25:50.069900] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:158) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1705031323033072531, v:0}, checkpoint_scn_in_ls_meta={val:1705031323033072531, v:0}, ls_id={id:1001}, service_type=“TRANS_SERVICE”)
[2024-01-12 15:25:50.069960] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1544) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=54][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:“10.0.104.30:2882”, has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0})
[2024-01-12 15:25:50.069979] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001})
[2024-01-12 15:25:50.069991] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [13110][T1004_TxCkpt][T1004][Y0-0000000000000000-0-0] [lt=10] succeed to update_clog_checkpoint(ret=0, ls_cnt=2)
[2024-01-12 15:25:50.086059] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:537) [12651][T1002_LSArchive][T1002][YB420A00681E-00060EB76EA1C8DD-0-0] [lt=20] gc stale ls task succ
[2024-01-12 15:25:50.088392] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [13109][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1004, server_version={version:{val:1705044349384431558, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=703954, in_cluster_service=true, cluster_version={val:1705044349384431557, v:0}, min_cluster_version={val:1705044349384431557, v:0}, max_cluster_version={val:1705044349384431557, v:0}, get_cluster_version_err=0, cluster_version_delta=703954, cluster_service_master=“10.0.104.30:2882”, cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=132690, succ_cluster_heartbeat_count=2, cluster_heartbeat_interval=50000, local_cluster_version={val:1705043767354547445, v:0}, local_cluster_delta=582733838, force_self_check=false, weak_read_refresh_interval=100000)
[2024-01-12 15:25:50.088428] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:688) [13109][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=28] no need to update min/max version(new_version={val:1705044349384431557, v:0}, min_version={val:1705044349384431557, v:0}, max_version={val:1705044349384431557, v:0})
[2024-01-12 15:25:50.103782] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [12643][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] query and update last id fail(ret=-4076, ret=“OB_NEED_WAIT”)
[2024-01-12 15:25:50.103951] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [12644][T1002_ReqMemEvi][T1002][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=6269, mem_used=68631552)
[2024-01-12 15:25:50.106396] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [13107][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret=“OB_NEED_WAIT”)
[2024-01-12 15:25:50.106525] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [12661][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}])
[2024-01-12 15:25:50.122785] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [11893][T1_L0_G0][T1][YB420A00681E-00060EB76C21E355-0-0] [lt=38] [RS_MGR] force set rs list(master_rs=“10.0.104.30:2882”)
[2024-01-12 15:25:50.122808] INFO [SERVER] handle_heartbeat (ob_service.cpp:3058) [11893][T1_L0_G0][T1][YB420A00681E-00060EB76C21E355-0-0] [lt=17] handle_heartbeat(ret=0, ret=“OB_SUCCESS”, hb_request={server:“10.0.104.30:2882”, server_id:1, rs_addr:“10.0.104.30:2882”, rs_server_status:1, epoch_id:1}, hb_response={zone:“zone1”, server:“10.0.104.30:2882”, sql_port:2881, build_version:“4.2.1.2_102000042023120514-ccdde7d34de421336c5362483d64bf2b73348bd4(Dec 5 2023 14:34:01)”, start_service_time:1705030830022445, server_health_status:{data_disk_status_:1, data_disk_status:“NORMAL”}}, time_cost=25)
[2024-01-12 15:25:50.123623] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1058) [11767][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self=“10.0.104.30:2882”, role=“LEADER”, total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3300, avg_group_log_size=330)
[2024-01-12 15:25:50.123691] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [11767][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=62] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:“10.0.104.30:2882”, has_set_deleted:false}, append size=3300)
[2024-01-12 15:25:50.133405] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:537) [13115][T1004_LSArchive][T1004][YB420A00681E-00060EB773F1C8DD-0-0] [lt=59] gc stale ls task succ
^C
[admin@ob30 log]$

#14

建议是内存的三倍值memory_limit
datafile_size数值可以少一点。后期如果想调小是不支持。只支持动态调整（调大）。

#15

我也是，卡半天了，也不提示失败，datafile_size给的是4T，等到明天会好吗？还是要怎么处理的？
看日志最后是报了4389的错误：
[2024-01-31 17:11:21.635403] INFO [LIB] stat (utility.h:1157) [21183][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=112, this=0x7f4a0951ee10, extra_info=“id:1”)
[2024-01-31 17:11:21.636920] INFO [LIB] stat (utility.h:1157) [21251][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=1402, this=0x7f4a09555330, extra_info=“palf_id:1”)
[2024-01-31 17:11:21.636956] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2129) [21251][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:“192.168.1.46:2882”, has_set_deleted:false}, accum_size=7706)
[2024-01-31 17:11:21.637007] INFO [LIB] stat (utility.h:1174) [21251][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=368, this=0x7f4a125fe570, extra_info="")
[2024-01-31 17:11:21.637059] INFO [LIB] stat (utility.h:1157) [21250][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=68] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=22, this=0x7f4a09555560, extra_info=“palf_id:1”)
[2024-01-31 17:11:21.644377] WDIAG [LIB] ~ObTimeGuard (utility.h:855) [21183][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] destruct(*this=time guard ‘[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1002, thread_timer_task’ cost too much time, used=10873, time_dist: do_cluster_heartbeat=10, generate_cluster_version=8743)

#16

目前的状态和错误日志麻烦发一下。
日志最好全一点还有配置信息。

#17

datafile_size 可以指定小一些 如 100G， datafile_maxsize 可以设置为 1T 、2T、4T 都可以。
datafile_size 设置小了不要紧，后面都可以改大。但是设置大了，后面就不能改小。

#18

我也遇到一样的问题，到了最后一步结束不了。日常使用，大部分功能没问题，但涉及到系统参数，集群详情的页面，都是报错。

#19

微信截图_20240112120048
微信截图_202401121200481371×1012 100 KB

微信截图_20240112120142
微信截图_202401121201421590×358 225 KB

微信截图_20240201105111
微信截图_202402011051111960×942 155 KB

#20

重新装datafile_size分配100G也不行，还是卡在那里，目前是测试服务器上开2个虚拟机（centos7），每个配置都是54G内存（资源有限，不做性能测试）、12T硬盘、12核CPU，2个可用区做集群（每台都是一个可用区）。

log.rar (7.5 MB)