Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Production-scale import for multi-GB CSV, JSON, and Parquet files

オープン
  • 何か問題がありますか？
#240 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
azure
領域
cli, databases

調査の方向性

Start by reading the existing import command and its parsing and write paths; the issue does not name specific files or tests. Compare the current streaming and serial-write behavior with the acceptance criteria, then identify the existing test entry points for import formats and error handling. Done means the documented formats, bounded concurrency, checkpoint/resume, progress, and failure output meet the criteria while existing JSON and CSV behavior remains compatible.

索引モデルが issue の本文から書いたものです。

説明

enhancement

Customer scenario

Importing a large local dataset (for example, a 10 GB CSV, JSON/JSONL, or Parquet file) into Azure Cosmos DB currently often pushes customers toward Spark or Databricks. That adds significant setup, learning, governance, and pricing overhead for what should be a straightforward data-loading workflow.

The shell should provide a production-ready experience comparable in simplicity to mongoimport: point the command at a file, connection/container, and relevant options, then let it safely and efficiently load the data.

Current behavior

The existing import command provides a useful foundation:

  • JSON Lines, JSON array, and CSV input
  • Streaming parsing rather than loading the entire file into memory
  • Insert and upsert modes
  • --dry-run validation
  • --continue-on-error
  • Partition-key mapping for CSV
  • Imported/failed counts and total RU charge

However, imports currently issue item writes serially. The command does not support Parquet, checkpoint/resume, configurable concurrency, durable failure output, or detailed progress reporting. As a result, it should not yet be positioned as an optimized or resilient multi-GB migration tool.

Proposed behavior

Extend import into a production-scale data loader while preserving its simple defaults.

Suggested capabilities:

  • Add Parquet input support.
  • Use bounded, configurable parallelism and Cosmos DB bulk execution where appropriate.
  • Respect throttling and retry guidance while allowing an optional RU or throughput budget.
  • Show periodic progress: records processed, succeeded, failed, elapsed time, throughput, RU consumed, and estimated completion when available.
  • Support checkpoints and resuming an interrupted import without starting from the beginning.
  • Write rejected records and structured error details to a user-selected file.
  • Support common field mapping and type-conversion needs, especially for CSV and Parquet.
  • Preserve streaming behavior and bounded memory usage for all formats.
  • Retain insert/upsert, dry-run, and continue-on-error behavior.
  • Produce both human-readable and structured summaries suitable for automation.

Possible usage:

import ./items.jsonl --mode=upsert --max-concurrency=16 --checkpoint=./items.checkpoint
import ./items.csv --partition-key=/tenantId --errors=./rejected.jsonl
import ./items.parquet --format=parquet --max-ru=50000

The exact option names are illustrative and should follow existing shell conventions.

Acceptance criteria

  • A multi-GB input file is processed with bounded memory usage.
  • JSONL, JSON array, and CSV behavior remains backward compatible.
  • Parquet files can be imported with documented type-mapping behavior.
  • Users can configure bounded write concurrency.
  • The importer handles service throttling without losing or silently skipping records.
  • An interrupted import can resume from a durable checkpoint.
  • Progress and final summaries include processed/succeeded/failed counts and observed RU charge.
  • Failed records can be written to a machine-readable file with actionable error details.
  • Cancellation leaves a valid checkpoint and does not report success.
  • Documentation clearly explains performance, consistency, idempotency, partition-key, and resume semantics.

Notes

This is intended for scalable client-side ingestion, not a transactional import. Writes across logical partitions cannot be made atomic as a single operation.

主要言語
C#
スター
4
フォーク
7
平均マージ
1日 21時間
マージ済み PR（30日）
30

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

Azure/CosmosDBShell のほかの issue

Azure/CosmosDBShell の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。