page-reduce: Phase 1 mis-types images and links, captures hidden content (eval on 30 sites)

Open
  • Is something wrong?
#563 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
javascript
Domain
tooling

Research direction

Start with plugins/web/cli/src/bundles/page-reduce.js and PHASE2-RULES.md; the issue links plugins/web/evals/page-reduce-skeleton-from-phase1 as an existing evaluation entry point. The bundle source lives in a separate internal repository, and the evaluation harness and per-site data are private, so first identify which fixes can be made in this repository. Done means addressing the applicable Phase 1 and Phase 2 findings and adding regression coverage for image, link, and hidden-content behavior.

Written by the indexing model from the issue text.

Description

bug

Summary

An evaluation of web page reduce and the reduce-page skill on 30 public sites (14 categories) found that Phase 1 mis-types images and links, captures hidden content, and splits complex pages unevenly. The skill's own bundle behaves the same as the CLI (checked on 3 sites), so the cause is in the shared bundle (plugins/web/cli/src/bundles/page-reduce.js). Phase 2 then cannot recover what Phase 1 lost.

Method: for each page, visible element counts and a full-page screenshot were captured from the live DOM, then compared with Phase 1 token counts (no LLM). Phase 2 was run by three model tiers (haiku, sonnet, opus) on 8 sites and scored blind by a judge against the screenshots.

Findings

Phase 1 (bundle and CLI)
  • 1. Images are almost never {IMAGE:WxH}. Median recall against visible images larger than 64px is 0. Photos become {ICON} (one travel page: 295 {ICON} tokens, 53 large images). All 39 {IMAGE} tokens in the run are {IMAGE:0x0}. Likely cause: tokenizeImages calls getBoundingClientRect() on a detached clone, which returns 0 (from reading the bundle, not yet confirmed by a patch).
  • 2. {LINK:label} is never emitted. Every <a href> becomes {CTA:label}: 0 {LINK} tokens on all 24 reduced sites. Likely cause: isCTALink compares getComputedStyle on detached clones.
  • 3. Hidden content is tokenized as visible. Mega-menu panels, collapsed sidebar trees (100+ links), mobile drawers, cookie panels and duplicate desktop/mobile copies appear in the output. Flagged on 8 of 8 judged sites.
  • 4. Card headlines and media are lost on card-heavy pages. Cards come out as {TEXT}{TEXT} or {CTA:title}. Heading recall against the live page is 0.03 to 0.17 on three news pages. Large hero and banner images are missing.
  • 5. Section detection is uneven. Counts range from 2 to 44. A news home page gets 3 sections (about 8 visual regions). Two commerce/SaaS pages get 44 and 42, many of them empty (carousel controls, modals, empty spans). Hero sections come out empty on a government page and a docs reference page.
  • 6. No output for simple pages. https://example.com and one plain forum page return EMPTY_RESULT after retries. A page with a single block should yield one section.
  • 7. Bot protection. 4 of 30 sites returned BLOCKED (corporate, e-commerce, education). Check whether web probe --save-recipe clears them, and document it in the skill.
Phase 2 (skill rules, PHASE2-RULES.md)
  • 8. {REPEAT} applied to non-identical siblings. The judge flagged {REPEAT:1} in all tiers. Note {REPEAT:1} is correct for exactly 3 identical siblings (keep 2, repeat 1), so only use on non-identical or fewer-than-3 groups is a defect. Rule 1 should state the minimum and what N counts.
  • 9. {NAV:N-items} is ambiguous. The rules do not say whether N is the total or the remainder. Tiers disagree. {NAV} and {REPEAT} are also swapped for link lists and content cards.
  • 10. {FORM:0-fields} for empty cookie panels, and empty sections kept. Add a rule to drop empty or hidden overlay sections.
  • 11. Non-identical siblings merged into one {REPEAT}, which drops headings and structure.
  • 12. Utility and design-system modifier classes kept (for example hds-*, state classes) while some semantic classes are stripped. Define which classes count as semantic.
  • 13. Output is not stable across models. One travel page produced 20 KB (opus), 51 KB (sonnet) and 143 KB (haiku). A size target or reduction check would help.

Blind judge means (1 to 5, 8 sites): layout 3.3 / 3.6 / 3.8, coverage 3.0 / 3.5 / 3.3, reduction 3.0 / 2.9 / 3.6, rule compliance 2.6 / 3.0 / 3.0 for haiku / sonnet / opus. Small n and one judge family, so indicative only.

What works

Heading, link and form recall are about 1.0 on text-first pages (wiki, docs, government, blog). 24 of 30 sites reduced.

Suggested direction

  1. Measure image size and link style on the live DOM (before cloning), or copy the measured values onto the clone.
  2. Skip elements that are not visible when tokenizing (display, visibility, size, off-screen drawers).
  3. Add a regression check: token counts versus live-DOM counts on a few fixtures (image recall, {LINK} present, hidden menu excluded). plugins/web/evals/page-reduce-skeleton-from-phase1 only covers Phase 2.
  4. Tighten PHASE2-RULES.md for items 8 to 12.

The bundle source is in the internal site-transfer-blueprint-detector repo (see plugins/web/docs/page-reduce-bundle.md), so fixes 1 to 5 land there and are copied here.

Reproduce

web page reduce <url> > out.json
jq -r '[.sections[].tokenizedHtml]|join("")' out.json | grep -o '{[A-Z]*[:}]' | sort | uniq -c

On image-heavy pages {IMAGE: is absent or 0x0, and no {LINK: appears.

Evaluation harness and per-site data live in a private repo (migration-lab, experiments/page-reduce-eval).

Dominant language
JavaScript
Stars
324
Forks
99
Avg merge
2d 7h
Merged PRs (30d)
121

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from adobe/skills

All issues in adobe/skills

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.