Credentials for different URLs isn't possible if they are under the same domain

Start by tracing the existing netrc authentication path and user-configuration handling, then review the linked buildstream-plugins-community merge request for related behavior. Done means credentials can be selected by URL globs, user configuration takes precedence over netrc, fallback remains compatible, and plugins can use the same mechanism.

Problem

We have a buildstream project that uses sources from private repositories.

These repositories require different credentials, but they are all under the same domain.
This is a problem, as netrc matches domain name but not url paths.

There needs to be a different way to authenticate based on URLs that isn't using netrc.

Potential solution

URLs could be matched based on globs. Services such as Gitlab lets you create tokens for a group, where any repositories under the group can use that token to pull the source or LFS tarballs.

These entries could be stored in user configuration, and I suggest prioritising authentication with it before using netrc as a fallback.
This wouldn't create compatibility issues, and allows for per-url authentication while still keeping a netrc file.

Example:

# URL will be matched against each entry. If URL matches an entry, those credentials are used.
authentication:
  # Match repository
  - url: "https://example.com/group1/repo1"
    username: "user1"
    password: "pass1"
  # Match repository group
  - url: "https://example.com/group1/*"
    username: "user2"
    password: "pass2"
  # Example of matching Gitlab LFS URL
  # Note that escape sequence is used to escape the ? character
  - url: 'https://example.com/api/v4/projects/group1%2Fnestedgroup%2Frepo/repository/files/lfs_object.tar.gz/raw\?ref=*&lfs=true'
    username: "user3"
    password: "user3"

Additional notes for plugins

This solution would allow for plugins to also authenticate using user configuration, without having to implement a custom solution for authenticating per-url.

The following merge request was created by me for buildstream-plugins-community, where I implement a solution similar to what I've described above, but credentials are obtained from ~/.gitconfig:

https://gitlab.com/BuildStream/buildstream-plugins-community/-/merge_requests/445.