Individual Firestore get() calls exceeding one second

Start with firestore.ts, getFirestore(), db.settings(), and the traced Firestore get() calls. Investigate whether the latency is caused by the Admin SDK, the gRPC connection pool, instrumentation-grpc, or expected Firestore behavior; done means identifying the source and documenting a supported way to reduce or assess the latency, if one exists.

• Operating System version: Google App Engine (Windows 10.0.19045 in dev)
• Firebase SDK version: firebase-admin@11.9.0
• Firebase Product: Firestore
• Node.js version: 16 (16.17.0 in dev)
• NPM version: 9.5.1

I use Firestore as my database for a simple backend server, and very slow reads are causing my API requests to take several seconds to complete.

My backend is an always-running GraphQL server running in Google App Engine, so it is not suffering from cold-start issues. The GAE instance and the selected Firestore instance are both running in US-central (nam5), so transfer delay should be negligible. Document sizes are 2-10KB, so large transfers should not be an issue. All other external REST calls complete in 50-120ms, so I don't believe it's a network saturation issue. My keys are generated UUIDs, so I don't believe it's a hotspotting issue.

My typical API processing makes 1-5 calls in parallel to documents to determine what the user is requesting, then gets 20-80 documents in parallel to build the response - i.e. a single waterfall. Using @opentelemetry/instrumentation-grpc@0.27.0, I am tracing these calls from within the server (and batching the traces). Typically the first batch of reads completes in 300-700ms, and the second batch takes 600-1200ms. This far exceeds what I would consider viable performance.

For the user to not be interrupted, I try to keep my P50 performance <300ms. With these numbers, I'm not even close. API calls that take 5+ seconds are not uncommon.

I feel like I've eliminated every possible variable, and all that is left is:
a) an issue with the node package, perhaps straining under 20+ simultaneous reads, or
b) misrepresented data coming from the instrumentation-grpc package, or
c) mismatched expectations of Firestore's capabilities. Unfortunately, it seems Firestore doesn't publish any SLAs or expectations. Is a single Firestore read taking 300ms expected, or an outlier needing investigation? I do see some reads happening in <40ms, which is far closer to my expectation.

Here's one of the simplest APIs I have, and it completes in 568ms. There are 56 reads across 6 batches. Shortest read time is 23ms (the first one), longest is 307ms. Most reads are >180ms.

Here's a taste of a more complex API that has 41 reads, and completes in 3298ms:

For reference, here's an API call that doesn't call Firestore. It completes in 9ms.

From these traces, it seems that the read times scales linearly with the number of reads in flight. Information online suggests that the firebase-admin package uses a GRPC pool under the hood, so I would expect the read time to be static until the GRPC connection pool is exhausted, then scale linearly after that point. Is there a way to configure or inspect what's happening with the GRPC connection pool?

Here's the code used to initialize the package.

firestore.ts

import {initializeApp} from 'firebase-admin/app';
import {CollectionReference, DocumentReference, FieldPath, getFirestore} from 'firebase-admin/firestore';

initializeApp();

export const db = getFirestore();
db.settings({ignoreUndefinedProperties: true, maxIdleChannels: 500});

TL;DR: How do I get my read times lower, or is this as good as I can expect them to be?