[🚀 Feature]: Remove dependencies on public URLs from tests

Aperta
#18,093 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@yashp676 ci sta già lavorando.

Dal 6/10/2026.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp, java, javascript, jquery, python, ruby, rust

Direzione di ricerca

Start with the failing Python targets in py/test/selenium/webdriver/common/bidi/browser_tests.py and _bidi/browser_tests.py, then review the other files listed in the issue by binding. Replace public test dependencies with local or reserved-host equivalents, remove the identified dead tests, and verify the affected suites run without public network access. Resolve the Rust mirror decision and audit requires-network tags before considering the cleanup complete.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

I-enhancement

Description

Several tests navigate to or connect to real public sites. These tests fail or flake when the site changes, is slow, is blocked by CI networking, or when the browser changes behavior on untrusted networks (we already have xfail_chrome markers for HTTP to HTTPS auto-upgrade on http://example.com). They also prevent running the suites with network access disabled.

A larger set of tests uses real registrable domains (example.com, google.com, test.com, etc.) purely as string data. Those never hit the network, but they are inconsistent and make it hard to audit which tests actually need network access.

This is breaking trunk CI now

Python / Browser Tests (chrome-bidi, windows) failed on trunk (4732af16c5). The only failures in the job are the two copies of one test, each failing all 3 attempts, so this is deterministic and not a flake:

  • //py:test/selenium/webdriver/common/bidi/browser_tests-chrome-bidi
  • //py:test/selenium/webdriver/common/_bidi/browser_tests-chrome-bidi
FAILED py\test\selenium\webdriver\common\bidi\browser_tests.py::test_create_user_context_with_direct_proxy[chrome]
AssertionError: assert 'example domain' in 'this domain is for use in documentation examples without needing permission. this is not a service, avoid relying on it for testing and monitoring purposes.\nlearn more'

Underlying cause: nothing changed in Selenium, the browser, or the driver. test_create_user_context_with_direct_proxy loads http://example.com/ and asserts the body text contains "example domain". IANA changed the page (Last-Modified: Mon, 28 Sep 2026 16:19:23 GMT, the day before the failing run) and removed the <h1>Example Domain</h1>. The body is now:

<body><p>This domain is for use in documentation examples without needing permission. This is not a service, avoid relying on it for testing and monitoring purposes.</p><a href=https://iana.org/help/example-domains>Learn more</a><script src=/s.js></script></body>

"Example Domain" survives only in <title>, which is not part of the body text. The new page text itself says not to rely on it for testing.

The immediate fix is to point this test at a page on the local test server. Matching the new text or the title would only reset the clock until the next change. The rest of this issue covers the other tests with the same exposure.

Tests that reach the public network

Binding Location Host Notes
Python py/test/selenium/webdriver/common/bidi/browser_tests.py (test_create_user_context_with_accept_insecure_certs, ..._with_direct_proxy, ..._with_proxy_and_accept_insecure_certs) self-signed.badssl.com, example.com Asserts on third-party page content ("example domain", badssl h1)
Python py/test/selenium/webdriver/common/_bidi/browser_tests.py same Near-duplicate of the file above; confirm whether both directories should exist
Python py/test/selenium/webdriver/common/bidi/browser_tests.py (..._manual_proxy_all_params) example.com Goes through the local fake proxy, but already xfail on Chrome because of HTTPS upgrade
Python py/test/selenium/webdriver/remote/remote_hub_connection_tests.py wrong.host.badssl.com Asserts on the TLS error text from a third-party host
Python py/test/selenium/webdriver/common/example2.py www.google.com Searches Google and asserts on results; nothing references it, candidate for deletion
.NET dotnet/test/webdriver/NetworkInterceptionTests.cs (TransformNetworkResponse) www.selenium.dev Response is replaced by the handler, so a local page works equally well
.NET dotnet/test/webdriver/BiDi/Input/CombinedInputActionsTests.cs (Paint) kleki.com [Test] is commented out; delete or port to a local page
Ruby rb/spec/integration/selenium/webdriver/manager_spec.rb saucelabs.com subdomains, watir.com, selenium.dev Guarded with skip_unless: {driver: :none} so they never run; they need subdomains or HTTPS
Rust rust/tests/mirror_tests.rs registry.npmmirror.com Third-party mirror; Selenium Manager tests need network by design, but this adds a dependency on a host we do not control
Shared pages common/src/web/mouse_interaction.html selenium.dev Loads jQuery and jQuery UI CSS/JS from selenium.dev instead of the local copies
Java java/test/org/openqa/selenium/PageLoadingTest.java www.thisurldoesnotexist.comx Not a reserved TLD; currently @NotYetImplemented for all browsers
Java java/test/org/openqa/selenium/grid/distributor/DistributorNodeAvailabilityTest.java does.not.exist Not a reserved TLD

Real domains used only as data (no network access)

Not failures today, but worth normalizing:

  • google.com: Java EndToEndTest and GridSupplier.main (spoofed session, never loads), JS test/lib/http_test.js, webdriver_test.js, until_test.js, Ruby IE options and Firefox profile specs, Python example2.py
  • test.com, some.host.com, cheese.com, my.domain.com, selenium2.ru, idp.com, foo.com, boo.com, yes.com, froogle.com, another.server.com, get.com, navigate-to.com, http_proxy.com, https_proxy.com, tracking.com, other.com, proxy.org, pac.url, a.base.url: mocks, proxy config, and parsing tests across all bindings
  • example.com / example.org in unit tests: reserved by RFC 2606 and acceptable as data, as long as nothing resolves or loads it
  • HttpClientTestBase.shouldAllowUrlsWithSchemesToBeUsed creates a client for http://example.com (the code comment already calls it a terrible choice); the request itself goes to the local server

Already correct, no change needed: *.test, *.example, and *.invalid hosts (not_a_valid_url.test, invalid-domain-that-does-not-exist.test, doesnotexist.invalid, rp.example, idp.example, grid.example), and URLs in comments, skip reasons, and error message assertions.

Proposed changes

  1. Serve everything from the local test web server.
    • Insecure cert tests: use the test server's existing self-signed HTTPS endpoint instead of badssl.com.
    • Proxy tests: navigate to a reserved hostname (for example http://proxied.test/) that only resolves via the fake proxy, and use a local page for the direct-proxy case.
    • Cookie domain and secure cookie tests: add subdomain and HTTPS support to the test environment (for example *.localhost aliases) so the Ruby specs can run instead of being permanently skipped.
    • TransformNetworkResponse: use a local page.
    • mouse_interaction.html: reference the jQuery files already in common/src/web.
  2. Replace unresolvable-host URLs with reserved names (.invalid or .test).
  3. Replace real domains used as data with reserved ones (example.com, *.example, *.test).
  4. Delete dead code: example2.py (and google_one_box if unused), the disabled Paint test.
  5. Decide on the Rust mirror test: keep with an explicit note that it depends on a third party, or point it at a mirror we control.
  6. Add a lint check that fails on new non-reserved hosts in test directories, with an allowlist for comments, bug tracker links, and Selenium Manager tests.
  7. Once clean, audit requires-network tags so only targets that truly need it (browser and driver downloads, Selenium Manager) keep it.

Have you considered any alternatives or workarounds?

Marking the affected tests as flaky or skipping them in CI hides the problem and loses coverage. Keeping public URLs but retrying does not help with content changes or blocked networks.

Lingua principale
Java
Stelle
34.6k
Fork
8.7k
Merge medio
1g 23h
PR unite (30g)
86

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SeleniumHQ/selenium

Tutte le issue di SeleniumHQ/selenium

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.