Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Production-scale import for multi-GB CSV, JSON, and Parquet files

未关闭
  • 哪里不对？
#240 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
25/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
azure
领域
cli, databases

调研方向

Start by reading the existing import command and its parsing and write paths; the issue does not name specific files or tests. Compare the current streaming and serial-write behavior with the acceptance criteria, then identify the existing test entry points for import formats and error handling. Done means the documented formats, bounded concurrency, checkpoint/resume, progress, and failure output meet the criteria while existing JSON and CSV behavior remains compatible.

由索引模型根据 Issue 内容生成。

描述

enhancement

Customer scenario

Importing a large local dataset (for example, a 10 GB CSV, JSON/JSONL, or Parquet file) into Azure Cosmos DB currently often pushes customers toward Spark or Databricks. That adds significant setup, learning, governance, and pricing overhead for what should be a straightforward data-loading workflow.

The shell should provide a production-ready experience comparable in simplicity to mongoimport: point the command at a file, connection/container, and relevant options, then let it safely and efficiently load the data.

Current behavior

The existing import command provides a useful foundation:

  • JSON Lines, JSON array, and CSV input
  • Streaming parsing rather than loading the entire file into memory
  • Insert and upsert modes
  • --dry-run validation
  • --continue-on-error
  • Partition-key mapping for CSV
  • Imported/failed counts and total RU charge

However, imports currently issue item writes serially. The command does not support Parquet, checkpoint/resume, configurable concurrency, durable failure output, or detailed progress reporting. As a result, it should not yet be positioned as an optimized or resilient multi-GB migration tool.

Proposed behavior

Extend import into a production-scale data loader while preserving its simple defaults.

Suggested capabilities:

  • Add Parquet input support.
  • Use bounded, configurable parallelism and Cosmos DB bulk execution where appropriate.
  • Respect throttling and retry guidance while allowing an optional RU or throughput budget.
  • Show periodic progress: records processed, succeeded, failed, elapsed time, throughput, RU consumed, and estimated completion when available.
  • Support checkpoints and resuming an interrupted import without starting from the beginning.
  • Write rejected records and structured error details to a user-selected file.
  • Support common field mapping and type-conversion needs, especially for CSV and Parquet.
  • Preserve streaming behavior and bounded memory usage for all formats.
  • Retain insert/upsert, dry-run, and continue-on-error behavior.
  • Produce both human-readable and structured summaries suitable for automation.

Possible usage:

import ./items.jsonl --mode=upsert --max-concurrency=16 --checkpoint=./items.checkpoint
import ./items.csv --partition-key=/tenantId --errors=./rejected.jsonl
import ./items.parquet --format=parquet --max-ru=50000

The exact option names are illustrative and should follow existing shell conventions.

Acceptance criteria

  • A multi-GB input file is processed with bounded memory usage.
  • JSONL, JSON array, and CSV behavior remains backward compatible.
  • Parquet files can be imported with documented type-mapping behavior.
  • Users can configure bounded write concurrency.
  • The importer handles service throttling without losing or silently skipping records.
  • An interrupted import can resume from a durable checkpoint.
  • Progress and final summaries include processed/succeeded/failed counts and observed RU charge.
  • Failed records can be written to a machine-readable file with actionable error details.
  • Cancellation leaves a valid checkpoint and does not report success.
  • Documentation clearly explains performance, consistency, idempotency, partition-key, and resume semantics.

Notes

This is intended for scalable client-side ingestion, not a transactional import. Writes across logical partitions cannot be made atomic as a single operation.

主要语言
C#
星标
4
派生
7
平均合并
2 天 11 分钟
30 天内合并 PR
28

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

Azure/CosmosDBShell 的其他 Issue

查看 Azure/CosmosDBShell 的全部 Issue

相似的 Issue

更多 C# Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。