【紧急】疑似OceanBase内置诊断SQL导致CPU满载

OceanBase
,
#1

【 使用版本 】OceanBase_CE_V4.2.1.7

我们正在使用 OMS 进行数据迁移 （从 MySQL 到 OceanBase），但是 OceanBase 中有一个完全没有参与到数据迁移的表，似乎被 OceanBase 自动执行了一些内置的诊断SQL，让 CPU 从 40+% 直接飙升到满载，产生了少则几十秒，多则几千秒的慢查询，也导致 OCP 一直频繁触发 CPU 使用率超限 的告警。

这些语句明显不是我们人为执行的。

类似的SQL语句比较多，并且每个SQL都只执行了一次，出于篇幅考虑，下面只列举两条（类似第二条的SQL非常多，只是分区名称或表名不同）：

SELECT /*+ NO_REWRITE USE_PLAN_CACHE(NONE) DBMS_STATS FULL(table) OPT_PARAM('ROWSETS_MAX_ROWS', 256) QUERY_TIMEOUT(14390919891)*/ COUNT(1), CALC_PARTITION_ID(`table`, SUBPART)
	, MAX(`id`), MIN(`id`)
	, COUNT(`id`), APPROX_COUNT_DISTINCT(`id`)
	, AVG(SYS_OP_OPNSIZE(`id`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`id`), MAX(`merchant_id`)
	, MIN(`merchant_id`), COUNT(`merchant_id`)
	, APPROX_COUNT_DISTINCT(`merchant_id`)
	, AVG(SYS_OP_OPNSIZE(`merchant_id`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`merchant_id`), MAX(`user_id`)
	, MIN(`user_id`), COUNT(`user_id`)
	, APPROX_COUNT_DISTINCT(`user_id`)
	, AVG(SYS_OP_OPNSIZE(`user_id`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`user_id`), MAX(`col1`)
	, MIN(`col1`), COUNT(`col1`)
	, APPROX_COUNT_DISTINCT(`col1`)
	, AVG(SYS_OP_OPNSIZE(`col1`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col1`), MAX(`col2`)
	, MIN(`col2`), COUNT(`col2`)
	, APPROX_COUNT_DISTINCT(`col2`)
	, AVG(SYS_OP_OPNSIZE(`col2`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col2`), MAX(`col0`)
	, MIN(`col0`), COUNT(`col0`)
	, APPROX_COUNT_DISTINCT(`col0`)
	, AVG(SYS_OP_OPNSIZE(`col0`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col0`), MAX(`col3`)
	, MIN(`col3`), COUNT(`col3`)
	, APPROX_COUNT_DISTINCT(`col3`)
	, AVG(SYS_OP_OPNSIZE(`col3`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col3`), MAX(`col4`)
	, MIN(`col4`), COUNT(`col4`)
	, APPROX_COUNT_DISTINCT(`col4`)
	, AVG(SYS_OP_OPNSIZE(`col4`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col4`), MAX(`col5`)
	, MIN(`col5`), COUNT(`col5`)
	, APPROX_COUNT_DISTINCT(`col5`)
	, AVG(SYS_OP_OPNSIZE(`col5`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col5`), MAX(`col6`)
	, MIN(`col6`), COUNT(`col6`)
	, APPROX_COUNT_DISTINCT(`col6`)
	, AVG(SYS_OP_OPNSIZE(`col6`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col6`), MAX(`wcol5`)
	, MIN(`wcol5`), COUNT(`wcol5`)
	, APPROX_COUNT_DISTINCT(`wcol5`)
	, AVG(SYS_OP_OPNSIZE(`wcol5`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`wcol5`), MAX(`status`)
	, MIN(`status`), COUNT(`status`)
	, APPROX_COUNT_DISTINCT(`status`)
	, AVG(SYS_OP_OPNSIZE(`status`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`status`), MAX(`col7`)
	, MIN(`col7`), COUNT(`col7`)
	, APPROX_COUNT_DISTINCT(`col7`)
	, AVG(SYS_OP_OPNSIZE(`col7`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col7`), MAX(`col8`)
	, MIN(`col8`), COUNT(`col8`)
	, APPROX_COUNT_DISTINCT(`col8`)
	, AVG(SYS_OP_OPNSIZE(`col8`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col8`), MAX(`col9`)
	, MIN(`col9`), COUNT(`col9`)
	, APPROX_COUNT_DISTINCT(`col9`)
	, AVG(SYS_OP_OPNSIZE(`col9`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col9`), MAX(`col10`)
	, MIN(`col10`), COUNT(`col10`)
	, APPROX_COUNT_DISTINCT(`col10`)
	, AVG(SYS_OP_OPNSIZE(`col10`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col10`), MAX(`col11`)
	, MIN(`col11`), COUNT(`col11`)
	, APPROX_COUNT_DISTINCT(`col11`)
	, AVG(SYS_OP_OPNSIZE(`col11`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col11`), MAX(`col12`)
	, MIN(`col12`), COUNT(`col12`)
	, APPROX_COUNT_DISTINCT(`col12`)
	, AVG(SYS_OP_OPNSIZE(`col12`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col12`), MAX(`col13`)
	, MIN(`col13`), COUNT(`col13`)
	, APPROX_COUNT_DISTINCT(`col13`)
	, AVG(SYS_OP_OPNSIZE(`col13`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col13`), MAX(`col14`)
	, MIN(`col14`), COUNT(`col14`)
	, APPROX_COUNT_DISTINCT(`col14`)
	, AVG(SYS_OP_OPNSIZE(`col14`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col14`), MAX(`col15`)
	, MIN(`col15`), COUNT(`col15`)
	, APPROX_COUNT_DISTINCT(`col15`)
	, AVG(SYS_OP_OPNSIZE(`col15`))
	, APPROX_COUNT_DISTINCT_SYNOPSIS(`col15`), ?
FROM `bj_game`.`table` PARTITION (`P20240707_0sp0`, 
`P20240707_0sp1`, 
`P20240707_0sp2`, 
`P20240707_0sp3`, 
`P20240707_0sp4`, 
`P20240707_0sp5`, 
`P20240707_0sp6`, 
`P20240707_0sp7`, 
`P20240707_0sp8`, 
`P20240707_0sp9`, 
`P20240707_0sp10`, 
`P20240707_0sp11`, 
`P20240707_0sp12`, 
`P20240707_0sp13`, 
`P20240707_0sp14`, 
`P20240707_0sp15`, 
`P20240707_0sp16`, 
`P20240707_0sp17`, 
`P20240707_0sp18`, 
`P20240707_0sp19`, 
`P20240707_0sp20`, 
`P20240707_0sp21`, 
`P20240707_0sp22`, 
`P20240707_0sp23`, 
`P20240707_0sp24`, 
`P20240707_0sp25`, 
`P20240707_0sp26`, 
`P20240707_0sp27`, 
`P20240707_0sp28`, 
`P20240707_0sp29`, 
`P20240707_0sp30`, 
`P20240707_0sp31`, 
`P20240708_0sp0`, 
`P20240708_0sp1`, 
`P20240708_0sp2`, 
`P20240708_0sp3`, 
`P20240708_0sp4`, 
`P20240708_0sp5`, 
`P20240708_0sp6`, 
`P20240708_0sp7`, 
`P20240708_0sp8`, 
`P20240708_0sp9`, 
`P20240708_0sp10`, 
`P20240708_0sp11`, 
`P20240708_0sp12`, 
`P20240708_0sp13`, 
`P20240708_0sp14`, 
`P20240708_0sp15`, 
`P20240708_0sp16`, 
`P20240708_0sp17`, 
`P20240708_0sp18`, 
`P20240708_0sp19`, 
`P20240708_0sp20`, 
`P20240708_0sp21`, 
`P20240708_0sp22`, 
`P20240708_0sp23`, 
`P20240708_0sp24`, 
`P20240708_0sp25`, 
`P20240708_0sp26`, 
`P20240708_0sp27`, 
`P20240708_0sp28`, 
`P20240708_0sp29`, 
`P20240708_0sp30`, 
`P20240708_0sp31`, 
`P20240709_0sp0`, 
`P20240709_0sp1`, 
`P20240709_0sp2`, 
`P20240709_0sp3`, 
`P20240709_0sp4`, 
`P20240709_0sp5`, 
`P20240709_0sp6`, 
`P20240709_0sp7`, 
`P20240709_0sp8`, 
`P20240709_0sp9`, 
`P20240709_0sp10`, 
`P20240709_0sp11`, 
`P20240709_0sp12`, 
`P20240709_0sp13`, 
`P20240709_0sp14`, 
`P20240709_0sp15`, 
`P20240709_0sp16`, 
`P20240709_0sp17`, 
`P20240709_0sp18`, 
`P20240709_0sp19`, 
`P20240709_0sp20`, 
`P20240709_0sp21`, 
`P20240709_0sp22`, 
`P20240709_0sp23`, 
`P20240709_0sp24`, 
`P20240709_0sp25`, 
`P20240709_0sp26`, 
`P20240709_0sp27`, 
`P20240709_0sp28`, 
`P20240709_0sp29`, 
`P20240709_0sp30`, 
`P20240709_0sp31`, 
`P20240710_0sp0`, 
`P20240710_0sp1`, 
`P20240710_0sp2`, 
`P20240710_0sp3`, 
`P20240710_0sp4`, 
`P20240710_0sp5`, 
`P20240710_0sp6`, 
`P20240710_0sp7`, 
`P20240710_0sp8`, 
`P20240710_0sp9`, 
`P20240710_0sp10`, 
`P20240710_0sp11`, 
`P20240710_0sp12`, 
`P20240710_0sp13`, 
`P20240710_0sp14`, 
`P20240710_0sp15`, 
`P20240710_0sp16`, 
`P20240710_0sp17`, 
`P20240710_0sp18`, 
`P20240710_0sp19`, 
`P20240710_0sp20`, 
`P20240710_0sp21`, 
`P20240710_0sp22`, 
`P20240710_0sp23`, 
`P20240710_0sp24`, 
`P20240710_0sp25`, 
`P20240710_0sp26`, 
`P20240710_0sp27`, 
`P20240710_0sp28`, 
`P20240710_0sp29`, 
`P20240710_0sp30`, 
`P20240710_0sp31`, 
`P20240711_0sp0`, 
`P20240711_0sp1`, 
`P20240711_0sp2`, 
`P20240711_0sp3`, 
`P20240711_0sp4`, 
`P20240711_0sp5`, 
`P20240711_0sp6`, 
`P20240711_0sp7`, 
`P20240711_0sp8`, 
`P20240711_0sp9`, 
`P20240711_0sp10`, 
`P20240711_0sp11`, 
`P20240711_0sp12`, 
`P20240711_0sp13`, 
`P20240711_0sp14`, 
`P20240711_0sp15`, 
`P20240711_0sp16`, 
`P20240711_0sp17`, 
`P20240711_0sp18`, 
`P20240711_0sp19`, 
`P20240711_0sp20`, 
`P20240711_0sp21`, 
`P20240711_0sp22`, 
`P20240711_0sp23`, 
`P20240711_0sp24`, 
`P20240711_0sp25`, 
`P20240711_0sp26`, 
`P20240711_0sp27`, 
`P20240711_0sp28`, 
`P20240711_0sp29`, 
`P20240711_0sp30`, 
`P20240711_0sp31`, 
`P20240712_0sp0`, 
`P20240712_0sp1`, 
`P20240712_0sp2`, 
`P20240712_0sp3`, 
`P20240712_0sp4`, 
`P20240712_0sp5`, 
`P20240712_0sp6`, 
`P20240712_0sp7`, 
`P20240712_0sp8`, 
`P20240712_0sp9`, 
`P20240712_0sp10`, 
`P20240712_0sp11`, 
`P20240712_0sp12`, 
`P20240712_0sp13`, 
`P20240712_0sp14`, 
`P20240712_0sp15`, 
`P20240712_0sp16`, 
`P20240712_0sp17`, 
`P20240712_0sp18`, 
`P20240712_0sp19`, 
`P20240712_0sp20`, 
`P20240712_0sp21`, 
`P20240712_0sp22`, 
`P20240712_0sp23`, 
`P20240712_0sp24`, 
`P20240712_0sp25`, 
`P20240712_0sp26`, 
`P20240712_0sp27`, 
`P20240712_0sp28`, 
`P20240712_0sp29`, 
`P20240712_0sp30`, 
`P20240712_0sp31`, 
`P20240713_0sp0`, 
`P20240713_0sp1`, 
`P20240713_0sp2`, 
`P20240713_0sp3`, 
`P20240713_0sp4`, 
`P20240713_0sp5`, 
`P20240713_0sp6`, 
`P20240713_0sp7`, 
`P20240713_0sp8`, 
`P20240713_0sp9`, 
`P20240713_0sp10`, 
`P20240713_0sp11`, 
`P20240713_0sp12`, 
`P20240713_0sp13`, 
`P20240713_0sp14`, 
`P20240713_0sp15`, 
`P20240713_0sp16`, 
`P20240713_0sp17`, 
`P20240713_0sp18`, 
`P20240713_0sp19`, 
`P20240713_0sp20`, 
`P20240713_0sp21`, 
`P20240713_0sp22`, 
`P20240713_0sp23`, 
`P20240713_0sp24`, 
`P20240713_0sp25`, 
`P20240713_0sp26`, 
`P20240713_0sp27`, 
`P20240713_0sp28`, 
`P20240713_0sp29`, 
`P20240713_0sp30`, 
`P20240713_0sp31`, 
`P20240714_0sp0`, 
`P20240714_0sp1`, 
`P20240714_0sp2`, 
`P20240714_0sp3`, 
`P20240714_0sp4`, 
`P20240714_0sp5`, 
`P20240714_0sp6`, 
`P20240714_0sp7`, 
`P20240714_0sp8`, 
`P20240714_0sp9`, 
`P20240714_0sp10`, 
`P20240714_0sp11`, 
`P20240714_0sp12`, 
`P20240714_0sp13`, 
`P20240714_0sp14`, 
`P20240714_0sp15`, 
`P20240714_0sp16`, 
`P20240714_0sp17`, 
`P20240714_0sp18`, 
`P20240714_0sp19`, 
`P20240714_0sp20`, 
`P20240714_0sp21`, 
`P20240714_0sp22`, 
`P20240714_0sp23`, 
`P20240714_0sp24`, 
`P20240714_0sp25`, 
`P20240714_0sp26`, 
`P20240714_0sp27`, 
`P20240714_0sp28`, 
`P20240714_0sp29`, 
`P20240714_0sp30`, 
`P20240714_0sp31`, 
`P20240715_0sp0`, 
`P20240715_0sp1`, 
`P20240715_0sp2`, 
`P20240715_0sp3`, 
`P20240715_0sp4`, 
`P20240715_0sp5`, 
`P20240715_0sp6`, 
`P20240715_0sp7`, 
`P20240715_0sp8`, 
`P20240715_0sp9`, 
`P20240715_0sp10`, 
`P20240715_0sp11`, 
`P20240715_0sp12`, 
`P20240715_0sp13`, 
`P20240715_0sp14`, 
`P20240715_0sp15`, 
`P20240715_0sp16`, 
`P20240715_0sp17`, 
`P20240715_0sp18`, 
`P20240715_0sp19`, 
`P20240715_0sp20`, 
`P20240715_0sp21`, 
`P20240715_0sp22`, 
`P20240715_0sp23`, 
`P20240715_0sp24`, 
`P20240715_0sp25`, 
`P20240715_0sp26`, 
`P20240715_0sp27`, 
`P20240715_0sp28`, 
`P20240715_0sp29`, 
`P20240715_0sp30`, 
`P20240715_0sp31`, 
`P20240716_0sp0`, 
`P20240716_0sp1`, 
`P20240716_0sp2`, 
`P20240716_0sp3`, 
`P20240716_0sp4`, 
`P20240716_0sp5`, 
`P20240716_0sp6`, 
`P20240716_0sp7`, 
`P20240716_0sp8`, 
`P20240716_0sp9`, 
`P20240716_0sp10`, 
`P20240716_0sp11`, 
`P20240716_0sp12`, 
`P20240716_0sp13`, 
`P20240716_0sp14`, 
`P20240716_0sp15`, 
`P20240716_0sp16`, 
`P20240716_0sp17`, 
`P20240716_0sp18`, 
`P20240716_0sp19`, 
`P20240716_0sp20`, 
`P20240716_0sp21`, 
`P20240716_0sp22`, 
`P20240716_0sp23`, 
`P20240716_0sp24`, 
`P20240716_0sp25`, 
`P20240716_0sp26`, 
`P20240716_0sp27`, 
`P20240716_0sp28`, 
`P20240716_0sp29`, 
`P20240716_0sp30`, 
`P20240716_0sp31`, 
`P20240717_0sp0`, 
`P20240717_0sp1`, 
`P20240717_0sp2`, 
`P20240717_0sp3`, 
`P20240717_0sp4`, 
`P20240717_0sp5`, 
`P20240717_0sp6`, 
`P20240717_0sp7`, 
`P20240717_0sp8`, 
`P20240717_0sp9`, 
`P20240717_0sp10`, 
`P20240717_0sp11`, 
`P20240717_0sp12`, 
`P20240717_0sp13`, 
`P20240717_0sp14`, 
`P20240717_0sp15`, 
`P20240717_0sp16`, 
`P20240717_0sp17`, 
`P20240717_0sp18`, 
`P20240717_0sp19`, 
`P20240717_0sp20`, 
`P20240717_0sp21`, 
`P20240717_0sp22`, 
`P20240717_0sp23`, 
`P20240717_0sp24`, 
`P20240717_0sp25`, 
`P20240717_0sp26`, 
`P20240717_0sp27`, 
`P20240717_0sp28`, 
`P20240717_0sp29`, 
`P20240717_0sp30`, 
`P20240717_0sp31`, 
`P20240718_0sp0`, 
`P20240718_0sp1`, 
`P20240718_0sp2`, 
`P20240718_0sp3`, 
`P20240718_0sp4`, 
`P20240718_0sp5`, 
`P20240718_0sp6`, 
`P20240718_0sp7`, 
`P20240718_0sp8`, 
`P20240718_0sp9`, 
`P20240718_0sp10`, 
`P20240718_0sp11`, 
`P20240718_0sp12`, 
`P20240718_0sp13`, 
`P20240718_0sp14`, 
`P20240718_0sp15`, 
`P20240718_0sp16`, 
`P20240718_0sp17`, 
`P20240718_0sp18`, 
`P20240718_0sp19`, 
`P20240718_0sp20`, 
`P20240718_0sp21`, 
`P20240718_0sp22`, 
`P20240718_0sp23`, 
`P20240718_0sp24`, 
`P20240718_0sp25`, 
`P20240718_0sp26`, 
`P20240718_0sp27`, 
`P20240718_0sp28`, 
`P20240718_0sp29`, 
`P20240718_0sp30`, 
`P20240718_0sp31`, 
`P20240719_0sp0`, 
`P20240719_0sp1`, 
`P20240719_0sp2`, 
`P20240719_0sp3`, 
`P20240719_0sp4`, 
`P20240719_0sp5`, 
`P20240719_0sp6`, 
`P20240719_0sp7`, 
`P20240719_0sp8`, 
`P20240719_0sp9`, 
`P20240719_0sp10`, 
`P20240719_0sp11`, 
`P20240719_0sp12`, 
`P20240719_0sp13`, 
`P20240719_0sp14`, 
`P20240719_0sp15`, 
`P20240719_0sp16`, 
`P20240719_0sp17`, 
`P20240719_0sp18`, 
`P20240719_0sp19`, 
`P20240719_0sp20`, 
`P20240719_0sp21`, 
`P20240719_0sp22`, 
`P20240719_0sp23`, 
`P20240719_0sp24`, 
`P20240719_0sp25`, 
`P20240719_0sp26`, 
`P20240719_0sp27`, 
`P20240719_0sp28`, 
`P20240719_0sp29`, 
`P20240719_0sp30`, 
`P20240719_0sp31`, 
`P20240720_0sp0`, 
`P20240720_0sp1`, 
`P20240720_0sp2`, 
`P20240720_0sp3`, 
`P20240720_0sp4`, 
`P20240720_0sp5`, 
`P20240720_0sp6`, 
`P20240720_0sp7`, 
`P20240720_0sp8`, 
`P20240720_0sp9`, 
`P20240720_0sp10`, 
`P20240720_0sp11`, 
`P20240720_0sp12`, 
`P20240720_0sp13`, 
`P20240720_0sp14`, 
`P20240720_0sp15`, 
`P20240720_0sp16`, 
`P20240720_0sp17`, 
`P20240720_0sp18`, 
`P20240720_0sp19`, 
`P20240720_0sp20`, 
`P20240720_0sp21`, 
`P20240720_0sp22`, 
`P20240720_0sp23`, 
`P20240720_0sp24`, 
`P20240720_0sp25`, 
`P20240720_0sp26`, 
`P20240720_0sp27`, 
`P20240720_0sp28`, 
`P20240720_0sp29`, 
`P20240720_0sp30`, 
`P20240720_0sp31`, 
`P20240721_0sp0`, 
`P20240721_0sp1`, 
`P20240721_0sp2`, 
`P20240721_0sp3`, 
`P20240721_0sp4`, 
`P20240721_0sp5`, 
`P20240721_0sp6`, 
`P20240721_0sp7`, 
`P20240721_0sp8`, 
`P20240721_0sp9`, 
`P20240721_0sp10`, 
`P20240721_0sp11`, 
`P20240721_0sp12`, 
`P20240721_0sp13`, 
`P20240721_0sp14`, 
`P20240721_0sp15`, 
`P20240721_0sp16`, 
`P20240721_0sp17`, 
`P20240721_0sp18`, 
`P20240721_0sp19`, 
`P20240721_0sp20`, 
`P20240721_0sp21`, 
`P20240721_0sp22`, 
`P20240721_0sp23`, 
`P20240721_0sp24`, 
`P20240721_0sp25`, 
`P20240721_0sp26`, 
`P20240721_0sp27`, 
`P20240721_0sp28`, 
`P20240721_0sp29`, 
`P20240721_0sp30`, 
`P20240721_0sp31`, 
`P20240722_0sp0`, 
`P20240722_0sp1`, 
`P20240722_0sp2`, 
`P20240722_0sp3`, 
`P20240722_0sp4`, 
`P20240722_0sp5`, 
`P20240722_0sp6`, 
`P20240722_0sp7`, 
`P20240722_0sp8`, 
`P20240722_0sp9`, 
`P20240722_0sp10`, 
`P20240722_0sp11`, 
`P20240722_0sp12`, 
`P20240722_0sp13`, 
`P20240722_0sp14`, 
`P20240722_0sp15`, 
`P20240722_0sp16`, 
`P20240722_0sp17`, 
`P20240722_0sp18`, 
`P20240722_0sp19`, 
`P20240722_0sp20`, 
`P20240722_0sp21`, 
`P20240722_0sp22`, 
`P20240722_0sp23`, 
`P20240722_0sp24`, 
`P20240722_0sp25`, 
`P20240722_0sp26`, 
`P20240722_0sp27`, 
`P20240722_0sp28`, 
`P20240722_0sp29`, 
`P20240722_0sp30`, 
`P20240722_0sp31`, 
`P20240723_0sp0`, 
`P20240723_0sp1`, 
`P20240723_0sp2`, 
`P20240723_0sp3`, 
`P20240723_0sp4`, 
`P20240723_0sp5`, 
`P20240723_0sp6`, 
`P20240723_0sp7`, 
`P20240723_0sp8`, 
`P20240723_0sp9`, 
`P20240723_0sp10`, 
`P20240723_0sp11`, 
`P20240723_0sp12`, 
`P20240723_0sp13`, 
`P20240723_0sp14`, 
`P20240723_0sp15`, 
`P20240723_0sp16`, 
`P20240723_0sp17`, 
`P20240723_0sp18`, 
`P20240723_0sp19`, 
`P20240723_0sp20`, 
`P20240723_0sp21`, 
`P20240723_0sp22`, 
`P20240723_0sp23`, 
`P20240723_0sp24`, 
`P20240723_0sp25`, 
`P20240723_0sp26`, 
`P20240723_0sp27`, 
`P20240723_0sp28`, 
`P20240723_0sp29`, 
`P20240723_0sp30`, 
`P20240723_0sp31`, 
`P20240724_0sp0`, 
`P20240724_0sp1`, 
`P20240724_0sp2`, 
`P20240724_0sp3`, 
`P20240724_0sp4`, 
`P20240724_0sp5`, 
`P20240724_0sp6`, 
`P20240724_0sp7`, 
`P20240724_0sp8`, 
`P20240724_0sp9`, 
`P20240724_0sp10`, 
`P20240724_0sp11`, 
`P20240724_0sp12`, 
`P20240724_0sp13`, 
`P20240724_0sp14`, 
`P20240724_0sp15`, 
`P20240724_0sp16`, 
`P20240724_0sp17`, 
`P20240724_0sp18`, 
`P20240724_0sp19`, 
`P20240724_0sp20`, 
`P20240724_0sp21`, 
`P20240724_0sp22`, 
`P20240724_0sp23`, 
`P20240724_0sp24`, 
`P20240724_0sp25`, 
`P20240724_0sp26`, 
`P20240724_0sp27`, 
`P20240724_0sp28`, 
`P20240724_0sp29`, 
`P20240724_0sp30`, 
`P20240724_0sp31`, 
`P20240725_0sp0`, 
`P20240725_0sp1`, 
`P20240725_0sp2`, 
`P20240725_0sp3`, 
`P20240725_0sp4`, 
`P20240725_0sp5`, 
`P20240725_0sp6`, 
`P20240725_0sp7`, 
`P20240725_0sp8`, 
`P20240725_0sp9`, 
`P20240725_0sp10`, 
`P20240725_0sp11`, 
`P20240725_0sp12`, 
`P20240725_0sp13`, 
`P20240725_0sp14`, 
`P20240725_0sp15`, 
`P20240725_0sp16`, 
`P20240725_0sp17`, 
`P20240725_0sp18`, 
`P20240725_0sp19`, 
`P20240725_0sp20`, 
`P20240725_0sp21`, 
`P20240725_0sp22`, 
`P20240725_0sp23`, 
`P20240725_0sp24`, 
`P20240725_0sp25`, 
`P20240725_0sp26`, 
`P20240725_0sp27`, 
`P20240725_0sp28`, 
`P20240725_0sp29`, 
`P20240725_0sp30`, 
`P20240725_0sp31`, 
`P20240726_0sp0`, 
`P20240726_0sp1`, 
`P20240726_0sp2`, 
`P20240726_0sp3`, 
`P20240726_0sp4`, 
`P20240726_0sp5`, 
`P20240726_0sp6`, 
`P20240726_0sp7`, 
`P20240726_0sp8`, 
`P20240726_0sp9`, 
`P20240726_0sp10`, 
`P20240726_0sp11`, 
`P20240726_0sp12`, 
`P20240726_0sp13`, 
`P20240726_0sp14`, 
`P20240726_0sp15`, 
`P20240726_0sp16`, 
`P20240726_0sp17`, 
`P20240726_0sp18`, 
`P20240726_0sp19`, 
`P20240726_0sp20`, 
`P20240726_0sp21`, 
`P20240726_0sp22`, 
`P20240726_0sp23`, 
`P20240726_0sp24`, 
`P20240726_0sp25`, 
`P20240726_0sp26`, 
`P20240726_0sp27`, 
`P20240726_0sp28`, 
`P20240726_0sp29`, 
`P20240726_0sp30`, 
`P20240726_0sp31`, 
`P20240727_0sp0`, 
`P20240727_0sp1`, 
`P20240727_0sp2`, 
`P20240727_0sp3`, 
`P20240727_0sp4`, 
`P20240727_0sp5`, 
`P20240727_0sp6`, 
`P20240727_0sp7`, 
`P20240727_0sp8`, 
`P20240727_0sp9`, 
`P20240727_0sp10`, 
`P20240727_0sp11`, 
`P20240727_0sp12`, 
`P20240727_0sp13`, 
`P20240727_0sp14`, 
`P20240727_0sp15`, 
`P20240727_0sp16`, 
`P20240727_0sp17`, 
`P20240727_0sp18`, 
`P20240727_0sp19`, 
`P20240727_0sp20`, 
`P20240727_0sp21`, 
`P20240727_0sp22`, 
`P20240727_0sp23`, 
`P20240727_0sp24`, 
`P20240727_0sp25`, 
`P20240727_0sp26`, 
`P20240727_0sp27`, 
`P20240727_0sp28`, 
`P20240727_0sp29`, 
`P20240727_0sp30`, 
`P20240727_0sp31`, 
`P20240728_0sp0`, 
`P20240728_0sp1`, 
`P20240728_0sp2`, 
`P20240728_0sp3`, 
`P20240728_0sp4`, 
`P20240728_0sp5`, 
`P20240728_0sp6`, 
`P20240728_0sp7`, 
`P20240728_0sp8`, 
`P20240728_0sp9`, 
`P20240728_0sp10`, 
`P20240728_0sp11`, 
`P20240728_0sp12`, 
`P20240728_0sp13`, 
`P20240728_0sp14`, 
`P20240728_0sp15`, 
`P20240728_0sp16`, 
`P20240728_0sp17`, 
`P20240728_0sp18`, 
`P20240728_0sp19`, 
`P20240728_0sp20`, 
`P20240728_0sp21`, 
`P20240728_0sp22`, 
`P20240728_0sp23`, 
`P20240728_0sp24`, 
`P20240728_0sp25`, 
`P20240728_0sp26`, 
`P20240728_0sp27`, 
`P20240728_0sp28`, 
`P20240728_0sp29`, 
`P20240728_0sp30`, 
`P20240728_0sp31`, 
`P20240729_0sp0`, 
`P20240729_0sp1`, 
`P20240729_0sp2`, 
`P20240729_0sp3`, 
`P20240729_0sp4`, 
`P20240729_0sp5`, 
`P20240729_0sp6`, 
`P20240729_0sp7`, 
`P20240729_0sp8`, 
`P20240729_0sp9`, 
`P20240729_0sp10`, 
`P20240729_0sp11`, 
`P20240729_0sp12`, 
`P20240729_0sp13`, 
`P20240729_0sp14`, 
`P20240729_0sp15`, 
`P20240729_0sp16`, 
`P20240729_0sp17`, 
`P20240729_0sp18`, 
`P20240729_0sp19`, 
`P20240729_0sp20`, 
`P20240729_0sp21`, 
`P20240729_0sp22`, 
`P20240729_0sp23`, 
`P20240729_0sp24`, 
`P20240729_0sp25`, 
`P20240729_0sp26`, 
`P20240729_0sp27`, 
`P20240729_0sp28`, 
`P20240729_0sp29`, 
`P20240729_0sp30`, 
`P20240729_0sp31`, 
`P20240730_0sp0`, 
`P20240730_0sp1`, 
`P20240730_0sp2`, 
`P20240730_0sp3`, 
`P20240730_0sp4`, 
`P20240730_0sp5`, 
`P20240730_0sp6`, 
`P20240730_0sp7`, 
`P20240730_0sp8`, 
`P20240730_0sp9`, 
`P20240730_0sp10`, 
`P20240730_0sp11`, 
`P20240730_0sp12`, 
`P20240730_0sp13`, 
`P20240730_0sp14`, 
`P20240730_0sp15`, 
`P20240730_0sp16`, 
`P20240730_0sp17`, 
`P20240730_0sp18`, 
`P20240730_0sp19`, 
`P20240730_0sp20`, 
`P20240730_0sp21`, 
`P20240730_0sp22`, 
`P20240730_0sp23`, 
`P20240730_0sp24`, 
`P20240730_0sp25`, 
`P20240730_0sp26`, 
`P20240730_0sp27`, 
`P20240730_0sp28`, 
`P20240730_0sp29`, 
`P20240730_0sp30`, 
`P20240730_0sp31`, 
`P20240611_0sp0`, 
`P20240611_0sp1`, 
`P20240611_0sp2`, 
`P20240611_0sp3`, 
`P20240611_0sp4`, 
`P20240611_0sp5`, 
`P20240611_0sp6`, 
`P20240611_0sp7`, 
`P20240611_0sp8`, 
`P20240611_0sp9`, 
`P20240611_0sp10`, 
`P20240611_0sp11`, 
`P20240611_0sp12`, 
`P20240611_0sp13`, 
`P20240611_0sp14`, 
`P20240611_0sp15`, 
`P20240611_0sp16`, 
`P20240611_0sp17`, 
`P20240611_0sp18`, 
`P20240611_0sp19`, 
`P20240611_0sp20`, 
`P20240611_0sp21`, 
`P20240611_0sp22`, 
`P20240611_0sp23`, 
`P20240611_0sp24`, 
`P20240611_0sp25`, 
`P20240611_0sp26`, 
`P20240611_0sp27`, 
`P20240611_0sp28`, 
`P20240611_0sp29`, 
`P20240611_0sp30`, 
`P20240611_0sp31`, 
`P20240612_0sp0`, 
`P20240612_0sp1`, 
`P20240612_0sp2`, 
`P20240612_0sp3`, 
`P20240612_0sp4`, 
`P20240612_0sp5`, 
`P20240612_0sp6`, 
`P20240612_0sp7`, 
`P20240612_0sp8`, 
`P20240612_0sp9`, 
`P20240612_0sp10`, 
`P20240612_0sp11`, 
`P20240612_0sp12`, 
`P20240612_0sp13`, 
`P20240612_0sp14`, 
`P20240612_0sp15`, 
`P20240612_0sp16`, 
`P20240612_0sp17`, 
`P20240612_0sp18`, 
`P20240612_0sp19`, 
`P20240612_0sp20`, 
`P20240612_0sp21`, 
`P20240612_0sp22`, 
`P20240612_0sp23`, 
`P20240612_0sp24`, 
`P20240612_0sp25`, 
`P20240612_0sp26`, 
`P20240612_0sp27`, 
`P20240612_0sp28`, 
`P20240612_0sp29`, 
`P20240612_0sp30`, 
`P20240612_0sp31`, 
`P20240613_0sp0`, 
`P20240613_0sp1`, 
`P20240613_0sp2`, 
`P20240613_0sp3`, 
`P20240613_0sp4`, 
`P20240613_0sp5`, 
`P20240613_0sp6`, 
`P20240613_0sp7`, 
`P20240613_0sp8`, 
`P20240613_0sp9`, 
`P20240613_0sp10`, 
`P20240613_0sp11`, 
`P20240613_0sp12`, 
`P20240613_0sp13`, 
`P20240613_0sp14`, 
`P20240613_0sp15`, 
`P20240613_0sp16`, 
`P20240613_0sp17`, 
`P20240613_0sp18`, 
`P20240613_0sp19`, 
`P20240613_0sp20`, 
`P20240613_0sp21`, 
`P20240613_0sp22`, 
`P20240613_0sp23`, 
`P20240613_0sp24`, 
`P20240613_0sp25`, 
`P20240613_0sp26`, 
`P20240613_0sp27`, 
`P20240613_0sp28`, 
`P20240613_0sp29`, 
`P20240613_0sp30`, 
`P20240613_0sp31`, 
`P20240614_0sp0`, 
`P20240614_0sp1`, 
`P20240614_0sp2`, 
`P20240614_0sp3`, 
`P20240614_0sp4`, 
`P20240614_0sp5`, 
`P20240614_0sp6`, 
`P20240614_0sp7`, 
`P20240614_0sp8`, 
`P20240614_0sp9`, 
`P20240614_0sp10`, 
`P20240614_0sp11`, 
`P20240614_0sp12`, 
`P20240614_0sp13`, 
`P20240614_0sp14`, 
`P20240614_0sp15`, 
`P20240614_0sp16`, 
`P20240614_0sp17`, 
`P20240614_0sp18`, 
`P20240614_0sp19`, 
`P20240614_0sp20`, 
`P20240614_0sp21`, 
`P20240614_0sp22`, 
`P20240614_0sp23`, 
`P20240614_0sp24`, 
`P20240614_0sp25`, 
`P20240614_0sp26`, 
`P20240614_0sp28`, 
`P20240614_0sp29`, 
`P20240614_0sp30`, 
`P20240614_0sp31`, 
`P20240615_0sp0`, 
`P20240615_0sp1`, 
`P20240615_0sp2`, 
`P20240615_0sp3`, 
`P20240615_0sp4`, 
`P20240615_0sp5`, 
`P20240615_0sp6`, 
`P20240615_0sp7`, 
`P20240615_0sp8`, 
`P20240615_0sp9`, 
`P20240615_0sp10`, 
`P20240615_0sp11`, 
`P20240615_0sp12`, 
`P20240615_0sp13`, 
`P20240615_0sp14`, 
`P20240615_0sp15`, 
`P20240615_0sp16`, 
`P20240615_0sp17`, 
`P20240615_0sp18`, 
`P20240615_0sp19`, 
`P20240615_0sp20`, 
`P20240615_0sp21`, 
`P20240615_0sp22`, 
`P20240615_0sp23`, 
`P20240615_0sp24`, 
`P20240615_0sp25`, 
`P20240615_0sp26`, 
`P20240615_0sp27`, 
`P20240615_0sp28`, 
`P20240615_0sp29`, 
`P20240615_0sp30`, 
`P20240615_0sp31`, 
`P20240616_0sp0`, 
`P20240616_0sp1`, 
`P20240616_0sp2`, 
`P20240616_0sp3`, 
`P20240616_0sp4`, 
`P20240616_0sp5`, 
`P20240616_0sp6`, 
`P20240616_0sp7`, 
`P20240616_0sp8`, 
`P20240616_0sp9`, 
`P20240616_0sp10`, 
`P20240616_0sp11`, 
`P20240616_0sp12`, 
`P20240616_0sp13`, 
`P20240616_0sp14`, 
`P20240616_0sp15`, 
`P20240616_0sp16`, 
`P20240616_0sp17`, 
`P20240616_0sp18`, 
`P20240616_0sp19`, 
`P20240616_0sp20`, 
`P20240616_0sp21`, 
`P20240616_0sp22`, 
`P20240616_0sp23`, 
`P20240616_0sp24`, 
`P20240616_0sp25`, 
`P20240616_0sp26`, 
`P20240616_0sp27`, 
`P20240616_0sp28`, 
`P20240616_0sp29`, 
`P20240616_0sp30`, 
`P20240616_0sp31`, 
`P20240617_0sp0`, 
`P20240617_0sp2`, 
`P20240617_0sp3`, 
`P20240617_0sp4`, 
`P20240617_0sp5`, 
`P20240617_0sp6`, 
`P20240617_0sp7`, 
`P20240617_0sp8`, 
`P20240617_0sp9`, 
`P20240617_0sp10`, 
`P20240617_0sp11`, 
`P20240617_0sp12`, 
`P20240617_0sp13`, 
`P20240617_0sp14`, 
`P20240617_0sp15`, 
`P20240617_0sp16`, 
`P20240617_0sp17`, 
`P20240617_0sp18`, 
`P20240617_0sp19`, 
`P20240617_0sp20`, 
`P20240617_0sp21`, 
`P20240617_0sp22`, 
`P20240617_0sp23`, 
`P20240617_0sp24`, 
`P20240617_0sp25`, 
`P20240617_0sp26`, 
`P20240617_0sp27`, 
`P20240617_0sp28`, 
`P20240617_0sp29`, 
`P20240617_0sp30`, 
`P20240617_0sp31`, 
`P20240619_0sp0`, 
`P20240619_0sp1`, 
`P20240619_0sp2`, 
`P20240619_0sp3`, 
`P20240619_0sp4`, 
`P20240619_0sp5`, 
`P20240619_0sp6`, 
`P20240619_0sp7`, 
`P20240619_0sp8`, 
`P20240619_0sp9`, 
`P20240619_0sp10`, 
`P20240619_0sp11`, 
`P20240619_0sp12`, 
`P20240619_0sp13`, 
`P20240619_0sp14`, 
`P20240619_0sp15`, 
`P20240619_0sp16`, 
`P20240619_0sp17`, 
`P20240619_0sp18`, 
`P20240619_0sp19`, 
`P20240619_0sp20`, 
`P20240619_0sp21`, 
`P20240619_0sp22`, 
`P20240619_0sp23`, 
`P20240619_0sp24`, 
`P20240619_0sp25`, 
`P20240619_0sp26`, 
`P20240619_0sp27`, 
`P20240619_0sp28`, 
`P20240619_0sp29`, 
`P20240619_0sp30`, 
`P20240619_0sp31`, 
`P20240620_0sp0`, 
`P20240620_0sp1`, 
`P20240620_0sp2`, 
`P20240620_0sp3`, 
`P20240620_0sp4`, 
`P20240620_0sp5`, 
`P20240620_0sp6`, 
`P20240620_0sp7`, 
`P20240620_0sp8`, 
`P20240620_0sp9`, 
`P20240620_0sp10`, 
`P20240620_0sp11`, 
`P20240620_0sp12`, 
`P20240620_0sp13`, 
`P20240620_0sp14`, 
`P20240620_0sp15`, 
`P20240620_0sp16`, 
`P20240620_0sp17`, 
`P20240620_0sp18`, 
`P20240620_0sp19`, 
`P20240620_0sp20`, 
`P20240620_0sp21`, 
`P20240620_0sp22`, 
`P20240620_0sp23`, 
`P20240620_0sp24`, 
`P20240620_0sp25`, 
`P20240620_0sp26`, 
`P20240620_0sp27`, 
`P20240620_0sp28`, 
`P20240620_0sp29`, 
`P20240620_0sp30`, 
`P20240620_0sp31`, 
`P20240630_0sp0`, 
`P20240630_0sp1`, 
`P20240630_0sp2`, 
`P20240630_0sp3`, 
`P20240630_0sp4`, 
`P20240630_0sp5`, 
`P20240630_0sp6`, 
`P20240630_0sp7`, 
`P20240630_0sp8`, 
`P20240630_0sp9`, 
`P20240630_0sp10`, 
`P20240630_0sp11`, 
`P20240630_0sp12`, 
`P20240630_0sp13`, 
`P20240630_0sp14`, 
`P20240630_0sp15`, 
`P20240630_0sp16`, 
`P20240630_0sp17`, 
`P20240630_0sp18`, 
`P20240630_0sp19`, 
`P20240630_0sp20`, 
`P20240630_0sp21`, 
`P20240630_0sp22`, 
`P20240630_0sp23`, 
`P20240630_0sp24`, 
`P20240630_0sp25`, 
`P20240630_0sp26`, 
`P20240630_0sp27`, 
`P20240630_0sp28`, 
`P20240630_0sp29`, 
`P20240630_0sp30`, 
`P20240630_0sp31`, 
`P20240701_0sp0`, 
`P20240701_0sp1`, 
`P20240701_0sp2`, 
`P20240701_0sp3`, 
`P20240701_0sp4`, 
`P20240701_0sp5`, 
`P20240701_0sp6`, 
`P20240701_0sp7`, 
`P20240701_0sp8`, 
`P20240701_0sp9`, 
`P20240701_0sp10`, 
`P20240701_0sp11`, 
`P20240701_0sp12`, 
`P20240701_0sp13`, 
`P20240701_0sp14`, 
`P20240701_0sp15`, 
`P20240701_0sp16`, 
`P20240701_0sp17`, 
`P20240701_0sp18`, 
`P20240701_0sp19`, 
`P20240701_0sp20`, 
`P20240701_0sp21`, 
`P20240701_0sp22`, 
`P20240701_0sp23`, 
`P20240701_0sp24`, 
`P20240701_0sp25`, 
`P20240701_0sp26`, 
`P20240701_0sp27`, 
`P20240701_0sp28`, 
`P20240701_0sp29`, 
`P20240701_0sp30`, 
`P20240701_0sp31`, 
`P20240702_0sp0`, 
`P20240702_0sp1`, 
`P20240702_0sp2`, 
`P20240702_0sp3`, 
`P20240702_0sp4`, 
`P20240702_0sp5`, 
`P20240702_0sp6`, 
`P20240702_0sp7`, 
`P20240702_0sp8`, 
`P20240702_0sp9`, 
`P20240702_0sp10`, 
`P20240702_0sp11`, 
`P20240702_0sp12`, 
`P20240702_0sp13`, 
`P20240702_0sp14`, 
`P20240702_0sp15`, 
`P20240702_0sp16`, 
`P20240702_0sp17`, 
`P20240702_0sp18`, 
`P20240702_0sp19`, 
`P20240702_0sp20`, 
`P20240702_0sp21`, 
`P20240702_0sp22`, 
`P20240702_0sp23`, 
`P20240702_0sp24`, 
`P20240702_0sp25`, 
`P20240702_0sp26`, 
`P20240702_0sp27`, 
`P20240702_0sp28`, 
`P20240702_0sp29`, 
`P20240702_0sp30`, 
`P20240702_0sp31`, 
`P20240703_0sp0`, 
`P20240703_0sp1`, 
`P20240703_0sp2`, 
`P20240703_0sp3`, 
`P20240703_0sp4`, 
`P20240703_0sp5`, 
`P20240703_0sp6`, 
`P20240703_0sp7`, 
`P20240703_0sp8`, 
`P20240703_0sp9`, 
`P20240703_0sp10`, 
`P20240703_0sp11`, 
`P20240703_0sp12`, 
`P20240703_0sp13`, 
`P20240703_0sp14`, 
`P20240703_0sp15`, 
`P20240703_0sp16`, 
`P20240703_0sp17`, 
`P20240703_0sp18`, 
`P20240703_0sp19`, 
`P20240703_0sp20`, 
`P20240703_0sp21`, 
`P20240703_0sp22`, 
`P20240703_0sp23`, 
`P20240703_0sp24`, 
`P20240703_0sp25`, 
`P20240703_0sp26`, 
`P20240703_0sp27`, 
`P20240703_0sp28`, 
`P20240703_0sp29`, 
`P20240703_0sp30`, 
`P20240703_0sp31`, 
`P20240704_0sp0`, 
`P20240704_0sp1`, 
`P20240704_0sp2`, 
`P20240704_0sp3`, 
`P20240704_0sp4`, 
`P20240704_0sp5`, 
`P20240704_0sp6`, 
`P20240704_0sp7`, 
`P20240704_0sp8`, 
`P20240704_0sp9`, 
`P20240704_0sp10`, 
`P20240704_0sp11`, 
`P20240704_0sp12`, 
`P20240704_0sp13`, 
`P20240704_0sp14`, 
`P20240704_0sp15`, 
`P20240704_0sp16`, 
`P20240704_0sp17`, 
`P20240704_0sp18`, 
`P20240704_0sp19`, 
`P20240704_0sp20`, 
`P20240704_0sp21`, 
`P20240704_0sp22`, 
`P20240704_0sp23`, 
`P20240704_0sp24`, 
`P20240704_0sp25`, 
`P20240704_0sp26`, 
`P20240704_0sp27`, 
`P20240704_0sp28`, 
`P20240704_0sp29`, 
`P20240704_0sp30`, 
`P20240704_0sp31`, 
`P20240705_0sp0`, 
`P20240705_0sp1`, 
`P20240705_0sp2`, 
`P20240705_0sp3`, 
`P20240705_0sp4`, 
`P20240705_0sp5`, 
`P20240705_0sp6`, 
`P20240705_0sp7`, 
`P20240705_0sp8`, 
`P20240705_0sp9`, 
`P20240705_0sp10`, 
`P20240705_0sp11`, 
`P20240705_0sp12`, 
`P20240705_0sp13`, 
`P20240705_0sp14`, 
`P20240705_0sp15`, 
`P20240705_0sp16`, 
`P20240705_0sp17`, 
`P20240705_0sp18`, 
`P20240705_0sp19`, 
`P20240705_0sp20`, 
`P20240705_0sp21`, 
`P20240705_0sp22`, 
`P20240705_0sp23`, 
`P20240705_0sp24`, 
`P20240705_0sp25`, 
`P20240705_0sp26`, 
`P20240705_0sp27`, 
`P20240705_0sp28`, 
`P20240705_0sp29`, 
`P20240705_0sp30`, 
`P20240705_0sp31`, 
`P20240706_0sp0`, 
`P20240706_0sp1`, 
`P20240706_0sp2`, 
`P20240706_0sp3`, 
`P20240706_0sp4`, 
`P20240706_0sp5`, 
`P20240706_0sp6`, 
`P20240706_0sp7`, 
`P20240706_0sp8`, 
`P20240706_0sp9`, 
`P20240706_0sp10`, 
`P20240706_0sp11`, 
`P20240706_0sp12`, 
`P20240706_0sp13`, 
`P20240706_0sp14`, 
`P20240706_0sp15`, 
`P20240706_0sp16`, 
`P20240706_0sp17`, 
`P20240706_0sp18`, 
`P20240706_0sp19`, 
`P20240706_0sp20`, 
`P20240706_0sp21`, 
`P20240706_0sp22`, 
`P20240706_0sp23`, 
`P20240706_0sp24`, 
`P20240706_0sp25`, 
`P20240706_0sp26`, 
`P20240706_0sp27`, 
`P20240706_0sp28`, 
`P20240706_0sp29`, 
`P20240706_0sp30`, 
`P20240706_0sp31`)
GROUP BY CALC_PARTITION_ID(`table`, SUBPART)

SELECT
/*+ NO_REWRITE USE_PLAN_CACHE(NONE) DBMS_STATS FULL(table) OPT_PARAM('ROWSETS_MAX_ROWS', 256) QUERY_TIMEOUT(14390928230)*/
TOP_K_FRE_HIST(?, `uid`, ?, ?)
FROM
`db`.`table` PARTITION (`P20240719_0`)

这种慢查询明显不正常，请问这种问题该如何解决呢 ？

2 个赞
#3

你贴的这两条SQL看着是统计信息收集的。用obdiag 巡检一下看看有没有啥问题。

  1. 安装obdiag; OceanBase分布式数据库-海量数据 笔笔算数
  2. 配置被诊断集群：obdiag config -hxx.xx.xx.xx -uroot@sys -p***** -Pxxx ,文档：OceanBase分布式数据库-海量数据 笔笔算数
  3. 执行巡检命令:obdiag check

先把巡检结果发出来看看

2 个赞
#4

好的，谢谢。我们先试试。

image
image1324×91 4.63 KB

不过这个 统计信息收集 的SQL占用的资源也太多了吧 ？最关键的一个SQL慢查询的时间超过了1个小时，CPU占用超过60% 。
不知道这个收集统计信息是怎么触发的，有什么 开关 或 触发时间 设置吗 ？
主要是担心，如果是生产环境，时不时的也这么搞一下，那就完蛋了。感觉就像一个定时炸弹一样。

2 个赞
#5

obdiag check 输出
obdiag_check_report_observer_2024-07-23-10-26-13.txt (21.0 KB)

2 个赞
#6

obdiag 巡检报告我看了下，除了一个内存模块超10GB的需要确认一下，其他的内容都还ok.

image
image1920×472 133 KB

你执行一下obdiag gather scene run --scene=observer.base ,生成的一份SQL文件你发回来我看一下。

PS: 统计信息的问题我们有专人提工单跟进一下

2 个赞
#7

obdiag gather scene run --scene=observer.base 输出, 请帮忙看下
sql_result.txt (1.4 MB)

1 个赞
#8

已联系统计信息相关研发同学分析中

1 个赞
#9

这些是统计信息收集相关的SQL，排查下最近一天是否有调度异常

– sys租户，查询非空则说明有租户调度异常（推荐）
SELECT tenant_id AS failed_scheduler_tenant_id
FROM oceanbase.__all_tenant t
WHERE NOT EXISTS(SELECT 1
FROM oceanbase.__all_virtual_task_opt_stat_gather_history h
WHERE TYPE = 1
AND start_time > date_sub(now(), interval 1 day)
AND h.tenant_id = t.tenant_id);

2 个赞
#10

查询结果为空

screenshot-20240723-181555
screenshot-20240723-181555709×259 33.7 KB

1 个赞
#11

说明统计信息的任务调度都是正常的，你这里cpu高可能是在业务运行期间同时运行了统计信息收集任务，导致cpu资源紧张。

继续排查下，
1.是否有统计信息收集失败的表：
–sys租户执行，可选指定相应租户id
SELECT t_opt.tenant_id,
t_opt.task_id,
task_opt.start_time AS task_start_time,
task_opt.end_time AS task_end_time,
d.database_name,
t.table_name,
t_opt.table_id,
t_opt.ret_code,
t_opt.start_time,
t_opt.end_time,
t_opt.memory_used,
t_opt.stat_refresh_failed_list,
t_opt.properties
FROM (
SELECT tenant_id,
task_id,
start_time,
end_time,
table_count
FROM oceanbase.__all_virtual_task_opt_stat_gather_history
WHERE type = 1
– AND tenant_id = {tenant_id}
AND start_time > date_sub(Now(), interval 1 day)) task_opt
JOIN oceanbase.__all_virtual_table_opt_stat_gather_history t_opt
JOIN oceanbase.__all_virtual_table t
JOIN oceanbase.__all_virtual_database d
WHERE t_opt.ret_code != 0
AND task_opt.task_id = t_opt.task_id
AND task_opt.tenant_id = t_opt.tenant_id
AND t_opt.tenant_id = t.tenant_id
AND t_opt.table_id = t.table_id
AND t.tenant_id = d.tenant_id
AND t.database_id = d.database_id
AND t_opt.table_id > 200000;

2.检查对应的调度的时间有无错乱等
–sys租户执行，可选指定相应租户id
SELECT tenant_id,
job_name,
what,
start_date,
this_date,
last_date,
next_date,
enabled
FROM oceanbase.__all_virtual_tenant_scheduler_job
WHERE tenant_id = {tenant_id}
AND job_name IN ( ‘MONDAY_WINDOW’, ‘TUESDAY_WINDOW’, ‘WEDNESDAY_WINDOW’,
‘THURSDAY_WINDOW’,
‘FRIDAY_WINDOW’, ‘SATURDAY_WINDOW’, ‘SUNDAY_WINDOW’ )
AND job != 0;

如果有大表自动统计信息收集时间过长或者失败则需要制定大表统计信息收集策略，

统计信息收集窗口也可根据业务情况调整到低峰期进行，参考

– MySQL业务租, 比如现在是2024-03-07, 周四早上11点, 需要调整为从周五凌晨2点开始:
call dbms_scheduler.set_attribute(‘MONDAY_WINDOW’, ‘NEXT_DATE’, ‘2024-03-11 02:00:00’);
call dbms_scheduler.set_attribute(‘TUESDAY_WINDOW’, ‘NEXT_DATE’, ‘2024-03-12 02:00:00’);
call dbms_scheduler.set_attribute(‘WEDNESDAY_WINDOW’, ‘NEXT_DATE’, ‘2024-03-13 02:00:00’);
call dbms_scheduler.set_attribute(‘THURSDAY_WINDOW’, ‘NEXT_DATE’, ‘2024-03-14 02:00:00’);
call dbms_scheduler.set_attribute(‘FRIDAY_WINDOW’, ‘NEXT_DATE’, ‘2024-03-08 02:00:00’);
call dbms_scheduler.set_attribute(‘SATURDAY_WINDOW’, ‘NEXT_DATE’, ‘2024-03-09 02:00:00’);
call dbms_scheduler.set_attribute(‘SUNDAY_WINDOW’, ‘NEXT_DATE’, ‘2024-03-10 02:00:00’);

–统计信息任务禁用，启用参考
– MySQL业务租户:
– 禁止
call dbms_scheduler.disable(‘MONDAY_WINDOW’);
call dbms_scheduler.disable(‘TUESDAY_WINDOW’);
call dbms_scheduler.disable(‘WEDNESDAY_WINDOW’);
call dbms_scheduler.disable(‘THURSDAY_WINDOW’);
call dbms_scheduler.disable(‘FRIDAY_WINDOW’);
call dbms_scheduler.disable(‘SATURDAY_WINDOW’);
call dbms_scheduler.disable(‘SUNDAY_WINDOW’);

– 启用
call dbms_scheduler.enable(‘MONDAY_WINDOW’);
call dbms_scheduler.enable(‘TUESDAY_WINDOW’);
call dbms_scheduler.enable(‘WEDNESDAY_WINDOW’);
call dbms_scheduler.enable(‘THURSDAY_WINDOW’);
call dbms_scheduler.enable(‘FRIDAY_WINDOW’);
call dbms_scheduler.enable(‘SATURDAY_WINDOW’);
call dbms_scheduler.enable(‘SUNDAY_WINDOW’);

#12

如果再次出现cpu负载高的情况，可以使用obdiag场景化信息收集
obdiag gather scene run --scene=observer.cpu_high

https://www.oceanbase.com/docs/common-obdiag-cn-1000000000902157

#13

存在一张统计信息收集失败的表.
result.txt (1.3 KB)

#14

好的, 非常感谢你的帮助, 我们调整下统计信息任务时间再观察下

#15

这个问题有再次出现吗？

#16

这种负载特别高的情况没有再次出现。
第一次估计是因为刚刚用 OMS 迁移了大量的数据到 OB，后面的数据变动就没有那么频繁了。
目前我们也调整了 维护窗口的任务执行时间。

不过，我的建议是，有没有办法将 收集统计信息 的 CPU 开销限制在一定范围内，比如不超过 30% 或 40%。
这样的话，能够让 OB 的负载更均衡，不容易出现负载大幅抖动的情况。