合并卡住了！！！！求救

yy_database · 2025 年3 月 31 日 14:22

【使用环境】
测试环境
【 OB or 其他组件】
OB 3-3-1
【使用版本】
oceanbase-4.2.5.2
【问题描述】
OB合并长时间未完成，附件为OBDIAG收集日志，请帮忙看一下什么原因，在线等！

obdiag_major_hold_20250331140141.zip (110.8 KB)

淇铭 · 2025 年3 月 31 日 15:16

查询一下这个信息
select * from CDB_OB_MAJOR_COMPACTION;

yy_database · 2025 年3 月 31 日 15:17

淇铭 · 2025 年3 月 31 日 15:20

直接查内部表__all_tablet_meta_table, 查询所有snapshot_version未达到指定版本号的tablet
select * from __all_tablet_meta_table where tenant_id=1002 and snapshot_version < 1743357604764449000;

yy_database · 2025 年3 月 31 日 15:24

我的版本里没有snapshot_version字段

淇铭 · 2025 年3 月 31 日 15:25

1、在RS leader所在的observer上去查看日志，基于’replica not merged’关键字。
grep “replica not merged” rootservice.log* | grep “T1002_MergeSche” | vim -

2、把这个时间2025.03.31 02:00:04所有节点的rootservice.log和observer.log 日志都发一下

淇铭 · 2025 年3 月 31 日 15:27

select * from __all_tablet_meta_table where tenant_id=1002 and compaction_scn < 1743357604764449000;

yy_database · 2025 年3 月 31 日 15:37

obclient(root@sys)[oceanbase]> select * from __all_tablet_meta_table where tenant_id=1002 and compaction_scn < 1743357604764449000;
Empty set (0.005 sec)

结果集为空

yy_database · 2025 年3 月 31 日 15:44

replica_not_merged.log (2.2 MB)
老师附件为grep “replica not merged”的内容。 2025.03.31 02:00:04时间点的observer.log和rootservice.log已经没有了，自动清理了

淇铭 · 2025 年3 月 31 日 16:33

sys租户下先切换到meta租户下查询信息
alter system change tenant META$1002;
select * from __all_tablet_meta_table where tenant_id=1002 and snapshot_version < 1743357604764449000;

yy_database · 2025 年3 月 31 日 16:35

obclient(root@sys)[(none)]> alter system change tenant META$1002;
ERROR 1235 (0A000): operation from proxy not supported
老师报错不让切换

淇铭 · 2025 年3 月 31 日 16:36

通过2881直连再查一下这个信息
alter system change tenant META$1002;
select * from __all_tablet_meta_table where tenant_id=1002 and tablet_id=200060;

淇铭 · 2025 年3 月 31 日 16:53

select * from __all_virtual_tablet_meta_table tenant_id=1002 and snapshot_version !=1743357604764449000;

select * from __all_virtual_compaction_diagnose_info where tenant_id=1002;
select * from __all_virtual_dag_warning_history where tenant_id=1002;

yy_database · 2025 年3 月 31 日 17:06

yy_database · 2025 年3 月 31 日 17:15

obclient(root@sys)[(none)]> select * from __all_virtual_tablet_meta_table where tenant_id=1002 and compaction_scn !=1743357604764449000;
Empty set (0.856 sec)

obclient(root@sys)[(none)]> select * from __all_virtual_compaction_diagnose_info where tenant_id=1002;
Empty set (0.292 sec)

obclient(root@sys)[oceanbase]> select * from __all_virtual_dag_warning_history where tenant_id=1002;
| 172.30.60.21 | 2882 | 1002 | YB42AC1E3C15-000630BAB2CCD6AE-0-0 | COMPACTION | MINI_MERGE | OB_SERVER_OUTOF_DISK_SPACE | WARNING | 2025-03-31 15:09:56.378658 | 2025-03-31 15:09:56.378658 | 0 | info=“DAG_MINI_MERGE”;ls_id=1003;tablet_id=1152921504606949297;compaction_scn=0;extra_info=“merge_type=“MINI_MERGE””; |
| 172.30.60.21 | 2882 | 1002 | YB42AC1E3C15-000630BAB2CCD6AF-0-0 | COMPACTION | MINI_MERGE | OB_SERVER_OUTOF_DISK_SPACE | WARNING | 2025-03-31 15:09:56.380246 | 2025-03-31 15:09:56.380246 | 0 | info=“DAG_MINI_MERGE”;ls_id=1003;tablet_id=221481;compaction_scn=0;extra_info=“merge_type=“MINI_MERGE””; |
| 172.30.60.21 | 2882 | 1002 | YB42AC1E3C15-000630BAB2CCD6B2-0-0 | COMPACTION | MINI_MERGE | OB_SERVER_OUTOF_DISK_SPACE | WARNING | 2025-03-31 15:09:56.384824 | 2025-03-31 15:09:56.384824 | 0 | info=“DAG_MINI_MERGE”;ls_id=1003;tablet_id=1152921504606949294;compaction_scn=0;extra_info=“merge_type=“MINI_MERGE””; |
| 172.30.60.21 | 2882 | 1002 | YB42AC1E3C15-000630BAB2CCD6B0-0-0 | COMPACTION | MINI_MERGE | OB_SERVER_OUTOF_DISK_SPACE | WARNING | 2025-03-31 15:09:56.390824 | 2025-03-31 15:09:56.390824 | 0 | info=“DAG_MINI_MERGE”;ls_id=1003;tablet_id=1152921504606949300;compaction_scn=0;extra_info=“merge_type=“MINI_MERGE””; |
| 172.30.60.21 | 2882 | 1002 | YB42AC1E3C15-000630BAB2CCD6B1-0-0 | COMPACTION | MINI_MERGE | OB_SERVER_OUTOF_DISK_SPACE | WARNING | 2025-03-31 15:09:56.393837 | 2025-03-31 15:09:56.393837 | 0 | info=“DAG_MINI_MERGE”;ls_id=1003;tablet_id=221475;compaction_scn=0;extra_info=“merge_type=“MINI_MERGE””; |
| 172.30.60.21 | 2882 | 1002 | YB42AC1E3C15-000630BAB2CD4CE2-0-0 | COMPACTION | MINOR_EXECUTE | OB_EAGAIN | WARNING | 2025-03-31 17:11:06.952745 | 2025-03-31 17:11:06.952745 | 0 | info=“DAG_MINOR_MERGE”;ls_id=1001;tablet_id=218938;compaction_scn=0;extra_info=“merge_type=“MINOR_MERGE””; |
| 172.30.60.21 | 2882 | 1002 | YB42AC1E3C15-000630BAB2CD4DC1-0-0 | COMPACTION | MINI_MERGE | OB_EAGAIN | WARNING | 2025-03-31 17:11:13.432499 | 2025-03-31 17:11:13.432499 | 0 | info=“DAG_MINI_MERGE”;ls_id=1001;tablet_id=1152921504606937032;compaction_scn=0;extra_info=“merge_type=“MINI_MERGE””;

yy_database · 2025 年3 月 31 日 17:15

老师 OB_SERVER_OUTOF_DISK_SPACE 是不是因为空间不足导致的啊？

淇铭 · 2025 年3 月 31 日 17:21

172.30.60.21在这个节点上，根据trace_id查找一下observer.log信息应该是空间不足导致的
grep “YB42AC1E3C15-000630BAB2CCD6AE-0-0” observer.*

淇铭 · 2025 年3 月 31 日 17:30

集群资源查询
select zone,concat(SVR_IP,’:’,SVR_PORT) observer,
cpu_capacity_max cpu_total,cpu_assigned_max cpu_assigned,
cpu_capacity-cpu_assigned_max as cpu_free,
round(memory_limit/1024/1024/1024,2) as memory_total,
round((memory_limit-mem_capacity)/1024/1024/1024,2) as system_memory,
round(mem_assigned/1024/1024/1024,2) as mem_assigned,
round((mem_capacity-mem_assigned)/1024/1024/1024,2) as memory_free,
round(log_disk_capacity/1024/1024/1024,2) as log_disk_capacity,
round(log_disk_assigned/1024/1024/1024,2) as log_disk_assigned,
round((log_disk_capacity-log_disk_assigned)/1024/1024/1024,2) as log_disk_free,
round((data_disk_capacity/1024/1024/1024),2) as data_disk,
round((data_disk_in_use/1024/1024/1024),2) as data_disk_used,
round((data_disk_capacity-data_disk_in_use)/1024/1024/1024,2) as data_disk_free
from oceanbase.gv$ob_servers;

租户资源查询
select a.zone,a.svr_ip,b.tenant_name,b.tenant_type, a.max_cpu, a.min_cpu,
round(a.memory_size/1024/1024/1024,2) memory_size_gb,
round(a.log_disk_size/1024/1024/1024,2) log_disk_size,
round(a.log_disk_in_use/1024/1024/1024,2) log_disk_in_use,
round(a.data_disk_in_use/1024/1024/1024,2) data_disk_in_use
from oceanbase.gv$ob_units a join oceanbase.dba_ob_tenants b on a.tenant_id=b.tenant_id order by b.tenant_name;

yy_database · 2025 年3 月 31 日 19:02

±------±------------------±----------±-------------±---------±-------------±--------------±-------------±------------±------------------±------------------±--------------±----------±---------------±---------------+
| zone | observer | cpu_total | cpu_assigned | cpu_free | memory_total | system_memory | mem_assigned | memory_free | log_disk_capacity | log_disk_assigned | log_disk_free | data_disk | data_disk_used | data_disk_free |
±------±------------------±----------±-------------±---------±-------------±--------------±-------------±------------±------------------±------------------±--------------±----------±---------------±---------------+
| zone1 | 172.30.60.11:2882 | 48 | 32 | 0 | 75.40 | 8.00 | 67.00 | 0.40 | 359.81 | 194.00 | 165.81 | 3217.78 | 2000.29 | 1217.49 |
| zone2 | 172.30.60.21:2882 | 48 | 32 | 0 | 75.40 | 8.00 | 67.00 | 0.40 | 359.81 | 194.00 | 165.81 | 4093.35 | 2021.98 | 2071.37 |
| zone3 | 172.30.60.41:2882 | 48 | 32 | 0 | 75.40 | 8.00 | 67.00 | 0.40 | 359.81 | 194.00 | 165.81 | 3120.55 | 1966.82 | 1153.73 |
±------±------------------±----------±-------------±---------±-------------±--------------±-------------±------------±------------------±------------------±--------------±----------±---------------±---------------+
3 rows in set (0.067 sec)

±------±-------------±------------±------------±--------±--------±---------------±--------------±----------------±-----------------+
| zone | svr_ip | tenant_name | tenant_type | max_cpu | min_cpu | memory_size_gb | log_disk_size | log_disk_in_use | data_disk_in_use |
±------±-------------±------------±------------±--------±--------±---------------±--------------±----------------±-----------------+
| zone1 | 172.30.60.11 | META$1002 | META | NULL | NULL | 6.20 | 18.60 | 14.85 | 0.70 |
| zone2 | 172.30.60.21 | META$1002 | META | NULL | NULL | 6.20 | 18.60 | 14.85 | 0.73 |
| zone3 | 172.30.60.41 | META$1002 | META | NULL | NULL | 6.20 | 18.60 | 14.85 | 0.73 |
| zone1 | 172.30.60.11 | obpaydb | USER | 28 | 28 | 55.80 | 167.40 | 133.87 | 1988.22 |
| zone2 | 172.30.60.21 | obpaydb | USER | 28 | 28 | 55.80 | 167.40 | 120.06 | 1880.94 |
| zone3 | 172.30.60.41 | obpaydb | USER | 28 | 28 | 55.80 | 167.40 | 133.87 | 1952.59 |
| zone1 | 172.30.60.11 | sys | SYS | 4 | 4 | 5.00 | 8.00 | 6.35 | 0.59 |
| zone2 | 172.30.60.21 | sys | SYS | 4 | 4 | 5.00 | 8.00 | 6.35 | 0.42 |
| zone3 | 172.30.60.41 | sys | SYS | 4 | 4 | 5.00 | 8.00 | 6.35 | 0.49 |
±------±-------------±------------±------------±--------±--------±---------------±--------------±----------------±-----------------+

感谢应该是磁盘原因我让系统刚把磁盘扩了

AntTech_ZFHDZ9 · 2025 年3 月 31 日 23:04

我之前也有一次是这样的，OB对磁盘的要求是有点高的。