Face Sensing loads its MediaPipe files from a root-relative path, and its CDN fallback doesn't run

Open
  • Is something wrong?
#717 0 comments 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
javascript, webpack
Domain
build-system, frontend

Research direction

Start with the detector setup in packages/scratch-vm/src/extensions/scratch3_face_sensing/index.js (around lines 94-110, where solutionPath is hard-coded), then compare with the publicPath comment in packages/scratch-gui/webpack.config.js (around lines 131-135). Check that the path resolves under a subdirectory and from file://, using the gui dev server. Done means the models load from the bundle location, and the fallback either goes or a load failure reaches the loading alert. The fallback choice is a maintainer decision, so confirm it before changing it.

Written by the indexing model from the issue text.

Description

Face Sensing creates its detector with a hard-coded, root-relative path:

https://github.com/scratchfoundation/scratch-editor/blob/develop/packages/scratch-vm/src/extensions/scratch3_face_sensing/index.js#L94-L110

solutionPath: '/chunks/mediapipe/face_detection' always resolves against the root of the site. The scratch-gui build copies those files next to its other chunks, and the library build uses publicPath: 'auto' so that chunks load relative to the bundle. The comment there says this is for hosts like scratch-desktop and scratch-android:

https://github.com/scratchfoundation/scratch-editor/blob/develop/packages/scratch-gui/webpack.config.js#L131-L135

Face Sensing doesn't follow that rule, so it may fail to load its models when the editor is:

  • hosted in a subdirectory (for example https://example.org/scratch/, where it requests https://example.org/chunks/...)
  • loaded from file://, where the path points at the root of the filesystem

scratch-android works around this by serving its web app at the root of its origin. Other hosts may not be able to.

The fallback to jsDelivr doesn't help in these cases. In testing on Android, the local load failed but createDetector didn't reject, so the .catch never ran and the "loading extension" alert stayed up. It looks like the files are only requested after createDetector resolves.

Possible fix
  • Build the path from webpack's runtime public path, for example __webpack_public_path__ + 'chunks/mediapipe/face_detection', so it resolves the same way as the gui's other chunks. This should be checked both in the prebuilt gui dist and for consumers that build scratch-vm from source through the webpack export condition.
  • Decide whether to keep the CDN fallback at all. If a host forgets to ship the model files, the fallback would hide that while online, and the problem would only show up offline or on networks that block the CDN. It also makes requests to a third party that the host app didn't choose. Failing visibly (for example, an error in the loading alert) may be easier to debug. If the fallback stays, the failure needs to be detected after createDetector resolves, or the fallback needs a timeout.
Testing
  • the gui dev server
  • the editor hosted in a subdirectory
  • scratch-desktop
  • scratch-android (it should keep working after the fix, with or without its root mapping)
Dominant language
JavaScript
Stars
310
Forks
299
Avg merge
19h 41m
Merged PRs (30d)
14

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from scratchfoundation/scratch-editor

All issues in scratchfoundation/scratch-editor

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.