meilisearch/MeiliSearch

Meilisearch service is unavailable three or four times a day

Open

#5236 opened on Jan 14, 2025

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Rust (20,887 stars) (733 forks)batch import
help wantedwindows

Description

Describe the bug Meilisearch service is deployed on my windows server. In the last half month, there will be three or four jams every day, each lasting several to ten minutes. During this period, the service is unavailable and no interface can be accessed, and then it will return to normal.

I created seven indexes on meilisearch, each with hundreds to hundreds of thousands of documents. I imported 30 million documents into one of the indexes half a month ago, and then this problem appeared, so I suspect that this problem is related to the amount of data.

To Reproduce After I imported 30 million documents, this problem will occur in the process of normal search using meilisearch.

My analysis In order to judge whether it is the server or the meilisearch service, I deployed another meilisearch service whit 7701 port on the server.Then I create a index and import twenty documents into it.

configuration:

  • My windows server: 64 cores 128G memory.
  • default meilisearch service: max_indexing_threads max_indexing_memory with deafult config.
  • 7701 meilisearch service: max_indexing_threads: 2; max_indexing_memory: 4 Gib.

Then I use jmeter to call their index interfaces twice a second. When the default meilisearch service is unavailable, the meilisearch service of 7701 is completely normal, and the cpu and memory of the server did not soar.

Expected behavior Don't be unavailable for a long time

Meilisearch version v1.11.0

Contributor guide

Tech stack
rustrest api
Domain
backenddatabase
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
RustMeilisearch internalsperformance debugging
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
10
Research direction
Investigate the root cause of periodic unavailability in Meilisearch after importing large indexes (30M documents). Review the indexing task scheduling and search handling code in the Meilisearch repository (likely in the index scheduler or milli crate). Consider the user's observation that a separate instance with limited threads/memory remains stable. Check if there is a resource contention issue or a bug in the search path when multiple indexes are queried. Look at relevant issues or PRs discussing similar large index problems. Files to examine: crate/index scheduler/src/lib.rs, crate/milli/src/store/mod.rs, and any Windows specific configuration.