【 使用环境 】生产环境
【 OB or 其他组件 】 OB
【 使用版本 】 4.2.1.3
【问题描述】 手动触发合并 超40分钟未合并完成 数据量显示 16G
【复现路径】问题出现前后相关操作
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接（右键跳转查看）：
unit 10核 12G ·1T express 显示内存占比 92%
50000一批 大约 70线程同步写入 先报 tenat 内存不足 然后手动触发合并
select * from __all_virtual_disk_stat; 看下是不是磁盘快满了导致没有空间合并了。
也可以使用最新的obdiag1.6版本 有专项合并检查。
https://www.oceanbase.com/softwarecenter
1 个赞
---------------±---------±--------------±-----------±--------------±--------------±--------------------±---------------+
| svr_ip | svr_port | total_size | used_size | free_size | is_disk_valid | disk_error_begin_ts | allocated_size |
±--------------±---------±--------------±-----------±--------------±--------------±--------------------±---------------+
| 10.164.47.178 | 2882 | 1099511627776 | 7172259840 | 1092335173632 | 1 | 0 | 1099511627776 |
| 10.164.47.175 | 2882 | 1099511627776 | 2789212160 | 1096718221312 | 1 | 0 | 1099511627776 |
| 10.164.47.176 | 2882 | 1099511627776 | 2795503616 | 1096711929856 | 1 | 0 | 1099511627776 |
| 10.164.47.179 | 2882 | 1099511627776 | 2795503616 | 1096711929856 | 1 | 0 | 1099511627776 |
| 10.164.47.177 | 2882 | 1099511627776 | 2793406464 | 1096714027008 | 1 | 0 | 1099511627776 |
±--------------±---------±--------------±-----------±--------------±--------------±--------------------±---------------+
磁盘够用的，可以先使用obdiag工具检查下 把结果附上 我们看下检查结果。
https://www.oceanbase.com/docs/common-obdiag-cn-1000000000564044
麻烦把这个目录打包一个包发到论坛上来看一下，里面是收集的一些详细的信息
麻烦到
这台机器上搜一下observer日志，grep “log disk space is almost full” 以及T1004
看起来弱读时间戳落后了
<
麻烦在178机器上搜一下observer日志，grep一下这个时间戳+1的值
弱读时间戳落后有可能是日志回放有问题，需要进一步排查一下
查一下show parameters like “%syslog%”;
麻烦看一下你的系统日志的设置，可能你的日志被回收了或者日志级别导致日志比较少
±------±---------±--------------±---------±----------------------------±----------±------±-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------±---------±--------±--------±------------------+
| zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
±------±---------±--------------±---------±----------------------------±----------±------±-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------±---------±--------±--------±------------------+
| zone1 | observer | 10.164.47.175 | 2882 | enable_syslog_recycle | NULL | True | specifies whether log file recycling is turned on. Value: True：turned on; False: turned off | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone1 | observer | 10.164.47.175 | 2882 | enable_syslog_wf | NULL | False | specifies whether any log message with a log level higher than ‘WARN’ would be printed into a separate file with a suffix of ‘wf’ | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone1 | observer | 10.164.47.175 | 2882 | enable_async_syslog | NULL | True | specifies whether use async log for observer.log, elec.log and rs.log | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone1 | observer | 10.164.47.175 | 2882 | max_syslog_file_count | NULL | 4 | specifies the maximum number of the log files that can co-exist before the log file recycling kicks in. Each log file can occupy at most 256MB disk space. When this value is set to 0, no log file will be removed. Range: [0, +∞) in integer | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone1 | observer | 10.164.47.175 | 2882 | diag_syslog_per_error_limit | NULL | 200 | DIAG syslog limitation for each error per second, exceeding syslog would be truncated | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone1 | observer | 10.164.47.175 | 2882 | syslog_io_bandwidth_limit | NULL | 30MB | Syslog IO bandwidth limitation, exceeding syslog would be truncated. Use 0 to disable ERROR log. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone1 | observer | 10.164.47.175 | 2882 | syslog_level | NULL | WDIAG | specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone4 | observer | 10.164.47.178 | 2882 | enable_syslog_recycle | NULL | True | specifies whether log file recycling is turned on. Value: True：turned on; False: turned off | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone4 | observer | 10.164.47.178 | 2882 | enable_syslog_wf | NULL | False | specifies whether any log message with a log level higher than ‘WARN’ would be printed into a separate file with a suffix of ‘wf’ | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone4 | observer | 10.164.47.178 | 2882 | enable_async_syslog | NULL | True | specifies whether use async log for observer.log, elec.log and rs.log | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone4 | observer | 10.164.47.178 | 2882 | max_syslog_file_count | NULL | 4 | specifies the maximum number of the log files that can co-exist before the log file recycling kicks in. Each log file can occupy at most 256MB disk space. When this value is set to 0, no log file will be removed. Range: [0, +∞) in integer | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone4 | observer | 10.164.47.178 | 2882 | diag_syslog_per_error_limit | NULL | 200 | DIAG syslog limitation for each error per second, exceeding syslog would be truncated | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone4 | observer | 10.164.47.178 | 2882 | syslog_io_bandwidth_limit | NULL | 30MB | Syslog IO bandwidth limitation, exceeding syslog would be truncated. Use 0 to disable ERROR log. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone4 | observer | 10.164.47.178 | 2882 | syslog_level | NULL | WDIAG | specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 10.164.47.176 | 2882 | enable_syslog_recycle | NULL | True | specifies whether log file recycling is turned on. Value: True：turned on; False: turned off | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 10.164.47.176 | 2882 | enable_syslog_wf | NULL | False | specifies whether any log message with a log level higher than ‘WARN’ would be printed into a separate file with a suffix of ‘wf’ | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 10.164.47.176 | 2882 | enable_async_syslog | NULL | True | specifies whether use async log for observer.log, elec.log and rs.log | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 10.164.47.176 | 2882 | max_syslog_file_count | NULL | 4 | specifies the maximum number of the log files that can co-exist before the log file recycling kicks in. Each log file can occupy at most 256MB disk space. When this value is set to 0, no log file will be removed. Range: [0, +∞) in integer | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 10.164.47.176 | 2882 | diag_syslog_per_error_limit | NULL | 200 | DIAG syslog limitation for each error per second, exceeding syslog would be truncated | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 10.164.47.176 | 2882 | syslog_io_bandwidth_limit | NULL | 30MB | Syslog IO bandwidth limitation, exceeding syslog would be truncated. Use 0 to disable ERROR log. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone2 | observer | 10.164.47.176 | 2882 | syslog_level | NULL | WDIAG | specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone5 | observer | 10.164.47.179 | 2882 | enable_syslog_recycle | NULL | True | specifies whether log file recycling is turned on. Value: True：turned on; False: turned off | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone5 | observer | 10.164.47.179 | 2882 | enable_syslog_wf | NULL | False | specifies whether any log message with a log level higher than ‘WARN’ would be printed into a separate file with a suffix of ‘wf’ | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone5 | observer | 10.164.47.179 | 2882 | enable_async_syslog | NULL | True | specifies whether use async log for observer.log, elec.log and rs.log | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone5 | observer | 10.164.47.179 | 2882 | max_syslog_file_count | NULL | 4 | specifies the maximum number of the log files that can co-exist before the log file recycling kicks in. Each log file can occupy at most 256MB disk space. When this value is set to 0, no log file will be removed. Range: [0, +∞) in integer | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone5 | observer | 10.164.47.179 | 2882 | diag_syslog_per_error_limit | NULL | 200 | DIAG syslog limitation for each error per second, exceeding syslog would be truncated | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone5 | observer | 10.164.47.179 | 2882 | syslog_io_bandwidth_limit | NULL | 30MB | Syslog IO bandwidth limitation, exceeding syslog would be truncated. Use 0 to disable ERROR log. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone5 | observer | 10.164.47.179 | 2882 | syslog_level | NULL | WDIAG | specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 10.164.47.177 | 2882 | enable_syslog_recycle | NULL | True | specifies whether log file recycling is turned on. Value: True：turned on; False: turned off | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 10.164.47.177 | 2882 | enable_syslog_wf | NULL | False | specifies whether any log message with a log level higher than ‘WARN’ would be printed into a separate file with a suffix of ‘wf’ | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 10.164.47.177 | 2882 | enable_async_syslog | NULL | True | specifies whether use async log for observer.log, elec.log and rs.log | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 10.164.47.177 | 2882 | max_syslog_file_count | NULL | 4 | specifies the maximum number of the log files that can co-exist before the log file recycling kicks in. Each log file can occupy at most 256MB disk space. When this value is set to 0, no log file will be removed. Range: [0, +∞) in integer | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 10.164.47.177 | 2882 | diag_syslog_per_error_limit | NULL | 200 | DIAG syslog limitation for each error per second, exceeding syslog would be truncated | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 10.164.47.177 | 2882 | syslog_io_bandwidth_limit | NULL | 30MB | Syslog IO bandwidth limitation, exceeding syslog would be truncated. Use 0 to disable ERROR log. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
| zone3 | observer | 10.164.47.177 | 2882 | syslog_level | NULL | WDIAG | specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
±------±---------±--------------±---------±----------------------------±----------±------±-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------±---------±--------±--------±------------------+
35 rows in set (0.011 sec)