Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Gemma 4 Multimodal & Advanced Features

Đang mở
  • Có gì không ổn?
#34 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
20/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Ít trao đổi
Công nghệ
kotlin
Lĩnh vực
ai, machine-learning

Hướng nghiên cứu

Start by inspecting the Gemma 4 runtime and Gemma4PromptFormatter, then trace how model configuration, tokenization, and inference are handled. The issue lists image and audio processors, vision and audio encoders, MoE routing, and LoRA ingestion as separate areas; choose one milestone and identify its existing entry points and tests. Done means meeting that milestone’s listed acceptance criteria with tests while keeping text-only behavior intact.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

Extend the Gemma 4 text-only runtime with multimodal capabilities (image, audio, video-via-frames), MoE support for the 26B A4B variant, and adapter/fine-tuning artifact ingestion. This issue covers vision/audio encoders, multimodal prompt handling, placeholder token replacement, and the expert routing layer.

Motivation

Gemma 4 is natively multimodal (text+image on all sizes; audio on E2B/E4B). Multimodal support unlocks on-device vision and audio use cases. The 26B A4B MoE variant offers a high-capability model with efficient active parameter usage, relevant for server-side deployments.

Functional Requirements

1. Image Processing & Vision Encoder
  • Implement image processor according to processor_config.json / preprocessor_config.json
  • Support variable Visual Token Budget (70 / 140 / 280 / 560 / 1120 tokens)
  • image_seq_length handling from config
  • Vision encoder inference (architecture per vision_config in config.json)
  • Placeholder token replacement: <|image|> tokens in the token stream are replaced by soft embeddings from the vision encoder output after tokenization
  • Multimodal content should appear before text in prompts for optimal performance
2. Audio Processing & Audio Encoder
  • Audio processor for E2B and E4B models (which support audio input)
  • Audio encoder inference (architecture per audio_config in config.json)
  • Placeholder token replacement: <|audio|> tokens replaced by audio encoder soft embeddings
  • Audio preprocessing pipeline (sampling rate, feature extraction)
3. Video Support (via Frames)
  • Video input decomposed into image frames
  • Frame extraction strategy (sampling rate, max frames)
  • Each frame processed through the vision encoder pipeline
  • Correct token budget allocation across frames
4. Multimodal Prompt Handling
  • Extend Gemma4PromptFormatter to handle interleaved text and media placeholders
  • Correct ordering: multimodal content before text in prompts
  • Token-index mapping for embedding insertion (map placeholder token positions to encoder output embeddings)
  • Support mixed-modality turns (image + text, audio + text)
5. MoE Dispatcher (26B A4B Variant)
  • Expert routing abstraction for Mixture-of-Experts layers
  • 26B A4B architecture: 128 total experts, 8 active + 1 shared expert per token
  • Top-K expert selection mechanism
  • Efficient expert weight loading (only active experts need to be in memory for inference)
  • Config-driven: MoE parameters from config.json (num_experts, num_active_experts, num_shared_experts)
6. Adapter / Fine-Tuning Artifact Ingestion
  • Support loading LoRA adapter weights alongside base model
  • Adapter weight merging or runtime application during inference
  • Compatibility with externally trained adapters (TRL, HF PEFT format)
7. Multimodal Data & Fine-Tuning Pipeline (Outline)
  • SFT dataset format specification for multimodal inputs: {image/audio/video frames + text prompt + text output}
  • Visual Token Budget variation in training data
  • Data governance: safety filtering, license checks, CSAM filtering compliance
  • Multimodal annotation pipeline requirements

Acceptance Criteria

Image Support
  • Image processor correctly preprocesses images according to processor_config.json
  • Visual Token Budget is configurable (70/140/280/560/1120) and correctly applied
  • Vision encoder produces embeddings of expected shape
  • <|image|> placeholder tokens are correctly replaced by vision encoder soft embeddings in the token stream
  • End-to-end image+text inference produces coherent outputs for E2B model
  • Multimodal content placed before text in prompts
Audio Support
  • Audio processor handles standard audio formats and sample rates
  • Audio encoder produces embeddings of expected shape
  • <|audio|> placeholder tokens are correctly replaced in the token stream
  • End-to-end audio+text inference works for E2B and E4B models
Video Support
  • Video frames are extracted and processed through the vision pipeline
  • Token budget is correctly distributed across frames
  • Multi-frame inference produces coherent outputs
MoE (26B A4B)
  • MoE config is parsed from config.json (expert counts, routing parameters)
  • Expert routing selects top-K experts per token correctly
  • Shared expert is always active alongside selected experts
  • 26B A4B model loads and runs inference end-to-end
  • Memory usage reflects active parameters (not total parameters)
Adapter Ingestion
  • LoRA adapter weights in HF PEFT format can be loaded alongside base Gemma 4 model
  • Adapter application produces different outputs compared to base model (verifiable)
General
  • All new features have unit and integration tests
  • CI passes with extended test suite
  • Text-only functionality remains unaffected by multimodal additions
  • Apache-2.0 license compliance maintained for all new components

Risks & Mitigations

Risk Mitigation
Vision/audio encoder complexity Start with image-only (simplest modality), validate end-to-end before adding audio
MoE memory requirements (26B total params) Implement lazy expert loading; defer MoE to later milestone if memory is prohibitive on target devices
Placeholder token replacement correctness Golden tests comparing embedding insertion positions against HF reference implementation
Multimodal data pipeline scope creep Keep fine-tuning pipeline as outline/spec only; implementation deferred to dedicated issue

Estimated Effort

Component Effort
Image Processor + Vision Encoder High
Audio Processor + Audio Encoder High
Video Frame Pipeline Medium
Multimodal Prompt Handling Medium
MoE Dispatcher High
Adapter Ingestion Medium
Tests & CI Medium

Milestones

  1. Image MVP — Single image + text inference for E2B (priority)
  2. Audio MVP — Audio + text inference for E2B/E4B
  3. MoE — 26B A4B loads and runs inference
  4. Video — Multi-frame video inference
  5. Adapters — LoRA weight loading and application
Ngôn ngữ chính
Kotlin
Star
8
Fork
3
Merge trung bình
1 ngày 16 giờ
Pull request đã merge (30 ngày)
48

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của SKaiNET-developers/SKaiNET-transformers

Tất cả issue của SKaiNET-developers/SKaiNET-transformers

Issue tương tự

Thêm issue về Kotlin

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.