sign/translate

[Feature] SMPL Texture Map Generator

オープン

#47 opened on 2022/07/05

 (0 件のコメント) (0 件のリアクション) (0 人の担当者)TypeScript (164 件のフォーク)auto 404
enhancementhelp wantedspoken-to-signed

Repository metrics

Stars
 (777 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Problem

Right now, the 3d avatar, and human realistic avatars have a specific appearance. We would like to support custom appearances.

Description

This feature will have an interface where the user can take a video of themselves (or a single picture?) and generate a SMPL texture map.

image

Here's an example of a texture map from MediaPipe face - https://github.com/apple2373/mediapipe-facemesh We could do something similar for hands and body using the "enable_segmentation" option.

MediPipe also supports hair segmentation, which may also be useful.

image

This can be done using a specific model, working on a single image, but probably best to compose on a video, prompting the user what we might be missing - hands, other side of face, etc.

Alternatives

Can use DensePose to generate SMPL maps, but that requires sending an image of the user to a server, and I don't think that's very good.

Or perhaps generate the full avatar, not only the texture map - https://github.com/sergeyprokudin/smplpix but that would even more likely require a server.

If we generate a SMPL/UV texture, it would be cross compatible. If we generate our own, whatever we can, it might be easier to create in the browser.

Additional context

It will support https://github.com/sign/translate/issues/33

コントリビューターガイド