docs: Document pattern for using non-exposed stream definitions as parent streams

Start with the YAML Reference and the linked Jira manifest examples, especially the definitions and streams sections showing __-prefixed parent streams. Document the internal-parent pattern, optional naming convention, exposure behavior, and mock-server testing implications; it is done when the advanced pattern is explained with a clear three-level example.

Summary

When building complex connectors with multi-level substream hierarchies, it's useful to define stream definitions that are only used internally as parent streams for other streams, without exposing them as top-level streams. This pattern is currently undocumented but is actively used in production connectors.

Problem

The current YAML Reference documentation explains that only entries in the top-level streams: array are exposed as runnable streams, but it doesn't explicitly document the pattern of:

1. Defining a full stream definition in definitions that is NOT listed in streams:
2. Using that definition solely as a parent_stream_config for another stream
3. The naming convention some connectors use (e.g., __ prefix) to signal "internal helper"

This pattern is particularly useful for 3-level nested substream hierarchies where an intermediate stream is needed to provide partition keys but shouldn't be exposed to users.

Example Implementation: Jira Connector

The Jira connector uses this pattern extensively. Here are code permalinks:

Internal/Private Stream Definitions (in definitions, NOT in streams:)

• __issue_property_keys_substream - Used as parent for issue_properties_stream
• __custom_issue_fields_substream - Used as parent for issue_custom_field_contexts
• __issue_custom_field_contexts_substream - Used as parent for issue_custom_field_options
• __boards_substream - Used as parent for board-related streams
• __story_points_issue_fields_substream - Used for story points configuration

How It's Used (3-level hierarchy example)

The issue_properties_stream references the internal __issue_property_keys_substream as its parent:

issue_properties_stream:
  # ...
  retriever:
    # ...
    partition_router:
      type: SubstreamPartitionRouter
      parent_stream_configs:
        - type: ParentStreamConfig
          stream: "#/definitions/__issue_property_keys_substream"  # <-- Internal stream reference

This creates a 3-level hierarchy:

1. issues_stream (grandparent - exposed)
2. __issue_property_keys_substream (parent - internal, NOT exposed)
3. issue_properties_stream (child - exposed)

Top-level streams: Section

The streams section only lists the streams that should be exposed to users - the __-prefixed definitions are intentionally omitted.

Suggested Documentation

Add a section to the YAML Reference or a new "Advanced Patterns" page that documents:

1. Pattern: Using stream definitions as internal parent streams
2. Use case: Multi-level substream hierarchies where intermediate streams shouldn't be exposed
3. Naming convention: The __ prefix convention (optional but recommended for clarity)
4. Behavior: Streams not listed in streams: will not be exposed by source.streams(config) - attempting to sync them will silently no-op
5. Testing implications: When writing mock server tests, always verify stream names against the streams: section to avoid testing non-existent streams

Context

This issue was discovered while creating comprehensive mock server tests for the Jira connector (airbytehq/airbyte#70884). The pattern caused confusion when attempting to test issue_property_keys as a stream, only to discover it's an internal-only definition.

---

Requested by: AJ Steers (@aaronsteers)
Related PR: https://github.com/airbytehq/airbyte/pull/70884
Devin session: https://app.devin.ai/sessions/f152f435f9d146688e476611ff864c30