[Feature]: Implement PaliGemma Vision-Language Model (Native PyTorch)

🚀 Feature Description

I propose implementing PaliGemma, a lightweight and powerful open Vision-Language Model (VLM) inspired by PaLI-3. This implementation will be a native PyTorch version within Kornia, avoiding external heavy dependencies like transformers.

PaliGemma combines: Vision Encoder: SigLIP (specifically the SigLip2 variant, which serves as the backbone). Language Decoder: Gemma (a decoder-only generic LLM). This feature will allow Kornia users to perform multimodal tasks (image + text $\to$ text) using official pre-trained weights.

📂 Feature Category

VLM/VLA Models (Vision Language Models/Agents) - Priority

💡 Motivation

Kornia currently lacks a native Vision-Language Model (VLM) pipeline, forcing users to rely on heavy external dependencies (like transformers) for multimodal tasks.

Implementing PaliGemma fills this gap by extending the existing SigLip2 backbone into a full VLM. This enables advanced tasks like Visual Question Answering (VQA) and captioning directly within Kornia, ensuring a unified, lightweight, and pure PyTorch workflow.

💭 Proposed Solution

I plan to implement the architecture in three stages, following a "pure PyTorch" approach:

Vision Backbone (SigLip2 Extension): Reuse and extend the existing SigLip2 implementation in Kornia to serve as the vision encoder.

Language Decoder (Gemma): Implement the Gemma decoder architecture from scratch using torch.nn modules (Attention, MLP, RMSNorm) to ensure no dependency on Hugging Face transformers.

Weight Loading Strategy: Create a mapping utility to load official pre-trained weights (from Google/HF) into the native Kornia model structure.

🔄 Alternatives Considered

No response

🎯 Use Cases

Visual Question Answering (VQA): Robots or agents analyzing a scene (e.g., "Is the safety valve open?").

Image Captioning: Automated reporting for surveillance or logging systems.

Object Detection/Segmentation: Generating bounding box coordinates or segmentation masks via text tokens.

Robotics: End-to-end vision-language planning where preprocessing speed is critical.

📝 Additional Context

No response

🤝 Contribution Intent

• I plan to submit a PR to implement this feature
• I'm requesting this feature but not planning to implement it