Skip to content
ClickHouse Docs
ClickHouse DocsClickHouse Docs
49.6kGet Started

DataStore 调试

View as MarkdownEdit this pageReport an issueSet up docs MCP

DataStore 提供了全面的调试工具，帮助您理解并优化数据管道。

调试工具概览

工具 用途 使用时机
explain() 查看执行计划 了解将执行哪些 SQL
Profiler 评估性能 找出耗时较长的操作
日志 查看执行细节 调试异常行为

快速决策矩阵

需求 工具 命令
查看执行计划 explain() ds.explain()
分析性能 Profiler config.enable_profiling()
调试 SQL 查询 日志 config.enable_debug()
以上全部 组合使用 见下文

快速设置

启用所有调试功能

from chdb import datastore as pd
from chdb.datastore.config import config

# Enable all debugging
config.enable_debug()        # Verbose logging
config.enable_profiling()    # Performance tracking

ds = pd.read_csv("data.csv")
result = ds.filter(ds['age'] > 25).groupby('city').agg({'salary': 'mean'})

# View execution plan
result.explain()

# Get profiler report
from chdb.datastore.config import get_profiler
profiler = get_profiler()
profiler.report()

explain() 方法

在运行查询前查看执行计划。

Querypython
ds = pd.read_csv("data.csv")

query = (ds
    .filter(ds['amount'] > 1000)
    .groupby('region')
    .agg({'amount': ['sum', 'mean']})
)

# View plan
query.explain()
Responsetext
Pipeline:
  Source: file('data.csv', 'CSVWithNames')
  Filter: amount > 1000
  GroupBy: region
  Aggregate: sum(amount), avg(amount)

Generated SQL:
SELECT region, SUM(amount) AS sum, AVG(amount) AS mean
FROM file('data.csv', 'CSVWithNames')
WHERE amount > 1000
GROUP BY region

详见explain() 文档。

性能分析

用于衡量每个操作的执行时间。

Querypython
from chdb.datastore.config import config, get_profiler

# Enable profiling
config.enable_profiling()

# Run operations
ds = pd.read_csv("large_data.csv")
result = (ds
    .filter(ds['amount'] > 100)
    .groupby('category')
    .agg({'amount': 'sum'})
    .sort('sum', ascending=False)
    .head(10)
    .to_df()
)

# View report
profiler = get_profiler()
profiler.report(min_duration_ms=0.1)
Responsetext
Performance Report
==================
Step                          Duration    Calls
----                          --------    -----
read_csv                      1.234s      1
filter                        0.002s      1
groupby                       0.001s      1
agg                           0.089s      1
sort                          0.045s      1
head                          0.001s      1
to_df (SQL execution)         0.567s      1
----                          --------    -----
Total                         1.939s      7

详见性能分析指南。

日志

查看详细的执行日志。

from chdb.datastore.config import config

# Enable debug logging
config.enable_debug()

# Run operations - logs will show:
# - SQL queries generated
# - Execution engine used
# - Cache hits/misses
# - Timing information

日志输出示例：

DEBUG - DataStore: Creating from file 'data.csv'
DEBUG - Query: SELECT region, SUM(amount) FROM ... WHERE amount > 1000 GROUP BY region
DEBUG - Engine: Using chdb for aggregation
DEBUG - Execution time: 0.089s
DEBUG - Cache: Storing result (key: abc123)

详见日志配置。

常见调试场景

1. 查询未返回预期结果

# Step 1: View the execution plan
query = ds.filter(ds['age'] > 25).groupby('city').sum()
query.explain(verbose=True)

# Step 2: Enable logging to see SQL
config.enable_debug()

# Step 3: Run and check logs
result = query.to_df()

2. 查询执行缓慢

# Step 1: Enable profiling
config.enable_profiling()

# Step 2: Run your query
result = process_data()

# Step 3: Check profiler report
profiler = get_profiler()
profiler.report()

# Step 4: Identify slow operations and optimize

3. 了解引擎选择

# Enable verbose logging
config.enable_debug()

# Run operations
result = ds.filter(ds['x'] > 10).apply(custom_func)

# Logs will show which engine was used for each operation:
# DEBUG - filter: Using chdb engine
# DEBUG - apply: Using pandas engine (custom function)

4. 调试缓存相关问题

# Enable debug to see cache operations
config.enable_debug()

# First run
result1 = ds.filter(ds['x'] > 10).to_df()
# LOG: Cache miss, executing query

# Second run (should use cache)
result2 = ds.filter(ds['x'] > 10).to_df()
# LOG: Cache hit, returning cached result

# If not caching when expected, check:
# - Are operations identical?
# - Is cache enabled? config.cache_enabled

最佳实践

1. 在开发环境中调试，而不要在生产环境中调试

# Development
config.enable_debug()
config.enable_profiling()

# Production
config.set_log_level(logging.WARNING)
config.set_profiling_enabled(False)

2. 执行大型查询前先使用 explain()

# Build query
query = ds.filter(...).groupby(...).agg(...)

# Check plan first
query.explain()

# If plan looks good, execute
result = query.to_df()

3. 优化前先做性能分析

# Don't guess what's slow - measure it
config.enable_profiling()
result = your_pipeline()
get_profiler().report()

4. 结果异常时检查 SQL

# View generated SQL
print(query.to_sql())

# Compare with expected SQL
# Run SQL directly in ClickHouse to verify

调试工具摘要

工具 命令 输出
执行计划说明 ds.explain() 执行步骤 + SQL
详细执行计划说明 ds.explain(verbose=True) + 元数据
查看 SQL ds.to_sql() SQL 查询字符串
启用调试 config.enable_debug() 详细日志
启用性能分析 config.enable_profiling() 耗时数据
Profiler 报告 get_profiler().report() 性能摘要
清空 Profiler get_profiler().reset() 清空耗时数据

后续步骤

Navigation