taosdata/TDengine

导入数据集时遇到了性能问题

Open

#34980 opened on Mar 29, 2026

View on GitHub
 (2 comments) (1 reaction) (0 assignees)C (24,849 stars) (5,002 forks)batch import
help wantedquestion

Description

我们在使用 TDengine 导入约 100GB 数据集时遇到了性能问题。服务器总内存为 198GB，但导入过程中内存占用长期维持在约 20GB，未能充分利用机器资源，导致整体导入速度较慢。

当前导入程序已经做了以下处理：

使用 JDBC 进行批量写入 关闭自动提交以减少事务开销 默认批大小为 10000 设置较长的 HTTP 连接与读写超时 尝试通过 ALTER ALL DNODES 'rpcQueueMemoryAllowed' ... 提高写入队列可用内存 时序数据采用多表 INSERT ... USING stable TAGS(...) VALUES(...) 模式写入

但实际效果上，TDengine 似乎仍然只使用了约 20GB 内存，导入吞吐没有随着可用内存提升而改善。

想请教：

这是否说明还有其他参数在限制写入性能？ 是否有更适合 100GB 级数据导入的推荐参数或工具？ 当前 JDBC 批量导入方案是否存在明显瓶颈？

Contributor guide

Tech stack
csql
Domain
databaseperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
TDengine architectureJDBC batch operationsDatabase performance tuningMemory management
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
10
Research direction
Investigate TDengine's memory allocation parameters, such as 'rpcQueueMemoryAllowed' and other memory limits. Review the official documentation for recommended configurations for importing 100GB datasets. Analyze the memory usage pattern using TDengine's monitoring tools or logs to identify bottlenecks in the current JDBC batch insert approach.