Sync from GitHub: e4ae36a6b560759a3e49020c0fe8fc46cedcea2d

README.md

@@ -1,14 +1,315 @@
----
-title: Rgbd Depth
-emoji: 🔥
-colorFrom: blue
-colorTo: indigo
-sdk: gradio
-sdk_version: 6.0.1
-app_file: app.py
-pinned: false
-license: apache-2.0
-short_description: Camera Depth Models for accurate metric depth estimation fro
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Camera Depth Models (CDM)
+
+Optimized Python package for RGB-D depth refinement using Vision Transformer encoders. This implementation is aligned with the [ByteDance CDM reference implementation](https://github.com/bytedance/camera-depth-models) with additional performance optimizations for CUDA, MPS (Apple Silicon), and CPU.
+
+[![Tests](https://github.com/Aedelon/camera-depth-models/actions/workflows/test.yml/badge.svg)](https://github.com/Aedelon/camera-depth-models/actions/workflows/test.yml)
+[![PyPI version](https://img.shields.io/pypi/v/rgbd-depth.svg)](https://pypi.org/project/rgbd-depth/)
+[![PyPI downloads](https://img.shields.io/pypi/dm/rgbd-depth.svg)](https://pypi.org/project/rgbd-depth/)
+[![Hugging Face Spaces](https://img.shields.io/badge/🤗%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/Aedelon/rgbd-depth)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![PyTorch 2.0+](https://img.shields.io/badge/PyTorch-2.0+-red.svg)](https://pytorch.org/)
+
+## 🎮 Try it Online
+
+[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/Aedelon/rgbd-depth)
+
+Try rgbd-depth directly in your browser with our interactive Gradio demo! No installation required.
+
+## Overview
+
+Camera Depth Models (CDMs) are sensor-specific depth models trained to produce clean, simulation-like depth maps from noisy real-world depth camera data. By bridging the visual gap between simulation and reality through depth perception, CDMs enable robotic policies trained purely in simulation to transfer directly to real robots.
+
+**Original work by ByteDance Research.** This package provides an optimized implementation with:
+- ✅ **Pixel-perfect alignment** with reference implementation (verified: 0 pixel difference)
+- ⚡ **Device-specific optimizations**: xFormers (CUDA), SDPA fallback, torch.compile
+- 🎯 **Mixed precision support**: FP16 (CUDA/MPS), BF16 (CUDA)
+- 🔧 **Better CLI**: Device selection, optimization control, precision modes
+- 📦 **Easy installation**: Single `pip install` command
+
+## Why This Package?
+
+This is an **optimized, production-ready** version of ByteDance's Camera Depth Models with several improvements:
+
+| Feature | ByteDance Original | This Package |
+|---------|-------------------|--------------|
+| **Installation** | Manual setup | `pip install rgbd-depth` |
+| **CUDA Optimization** | Basic | xFormers (~8% faster) + torch.compile |
+| **Apple Silicon (MPS)** | Not optimized | Native support with fallbacks |
+| **Mixed Precision** | Manual | Automatic FP16/BF16 with `--precision` flag |
+| **CLI** | Basic | Enhanced with device selection, optimization control |
+| **Documentation** | Minimal | Comprehensive guides (README + OPTIMIZATION.md) |
+| **Testing** | None | CI/CD with automated tests |
+| **PyPI Package** | No | ✅ Yes (`rgbd-depth`) |
+
+**Choose this package if you want:**
+- 🚀 Faster inference on CUDA (xFormers) or Apple Silicon (MPS)
+- 🎯 Easy mixed precision (FP16/BF16) without code changes
+- 📦 Simple installation via PyPI
+- 🔧 Production-ready CLI with device/precision control
+- ✅ Maintained with CI/CD and tests
+
+### Key Features
+
+- **Metric Depth Estimation**: Produces accurate absolute depth measurements in meters
+- **Multi-Camera Support**: Optimized models for various depth sensors (RealSense D405/D435/L515, ZED 2i, Azure Kinect)
+- **Performance Optimizations**: ~8% faster on CUDA with xFormers, automatic backend selection
+- **Mixed Precision**: FP16/BF16 support for faster inference on compatible hardware
+- **Sim-to-Real Ready**: Generates simulation-quality depth from real camera data
+
+## Architecture
+
+CDM uses a dual-branch Vision Transformer architecture:
+- **RGB Branch**: Extracts semantic information from RGB images
+- **Depth Branch**: Processes noisy depth sensor data
+- **Cross-Attention Fusion**: Combines RGB semantics with depth scale information
+- **DPT Decoder**: Produces final metric depth estimation
+
+Supported ViT encoder sizes:
+- `vits`: Small (64 features, 384 output channels)
+- `vitb`: Base (128 features, 768 output channels)
+- `vitl`: Large (256 features, 1024 output channels)
+- `vitg`: Giant (384 features, 1536 output channels)
+
+All pretrained models we provide are based on `vitl`.
+
+## Installation
+
+### From PyPI (recommended)
+
+```bash
+# Basic installation
+pip install rgbd-depth
+
+# With CUDA optimizations (xFormers)
+pip install rgbd-depth[xformers]
+
+# Development installation
+git clone https://github.com/Aedelon/camera-depth-models.git
+cd camera-depth-models
+pip install -e .
+```
+
+**Requirements:**
+- Python 3.8+
+- PyTorch 2.0+ with appropriate CUDA/MPS support
+- OpenCV, NumPy, Pillow
+
+## Quick Start
+
+```bash
+# CUDA (optimizations auto-enabled, FP16 for best speed)
+python infer.py --input rgb.png --depth depth.png --precision fp16
+
+# Apple Silicon (MPS)
+python infer.py --input rgb.png --depth depth.png --device mps
+
+# CPU (FP32 only)
+python infer.py --input rgb.png --depth depth.png --device cpu
+```
+
+> Example images are provided in `input_data/`. Pre-trained models can be downloaded from [Hugging Face](https://huggingface.co/collections/depth-anything/camera-depth-models-68b521181dedd223f4b020db).
+
+## Usage
+
+### Command Line Interface
+
+**Basic inference:**
+```bash
+python infer.py \
+    --input /path/to/rgb.png \
+    --depth /path/to/depth.png \
+    --output refined_depth.png
+```
+
+**CUDA with optimizations (default):**
+```bash
+# FP32 (best accuracy)
+python infer.py --input rgb.png --depth depth.png
+
+# FP16 (best speed, ~2× faster)
+python infer.py --input rgb.png --depth depth.png --precision fp16
+
+# BF16 (best stability)
+python infer.py --input rgb.png --depth depth.png --precision bf16
+
+# Disable optimizations (debugging)
+python infer.py --input rgb.png --depth depth.png --no-optimize
+```
+
+**Apple Silicon (MPS):**
+```bash
+# FP32 (default)
+python infer.py --input rgb.png --depth depth.png --device mps
+
+# FP16 (faster)
+python infer.py --input rgb.png --depth depth.png --device mps --precision fp16
+```
+
+**CPU:**
+```bash
+# FP32 only (FP16 not recommended on CPU)
+python infer.py --input rgb.png --depth depth.png --device cpu
+```
+
+### Command Line Arguments
+
+**Required:**
+- `--input`: Path to RGB input image (JPG/PNG)
+- `--depth`: Path to depth input image (PNG, 16-bit or 32-bit)
+
+**Optional:**
+- `--output`: Output visualization path (default: `output.png`)
+- `--device`: Device to use: `auto`, `cuda`, `mps`, `cpu` (default: `auto`)
+- `--precision`: Precision mode: `fp32`, `fp16`, `bf16` (default: `fp32`)
+- `--no-optimize`: Disable optimizations on CUDA (for debugging)
+- `--encoder`: Model size: `vits`, `vitb`, `vitl`, `vitg` (default: `vitl`)
+- `--input-size`: Input resolution for inference (default: 518)
+- `--depth-scale`: Scale factor for depth values (default: 1000.0)
+- `--max-depth`: Maximum valid depth in meters (default: 6.0)
+
+### Python API
+
+```python
+import torch
+from rgbddepth.dpt import RGBDDepth
+import cv2
+import numpy as np
+
+# Load model with optimizations
+model = RGBDDepth(encoder='vitl', features=256, use_xformers=True)
+model.load_state_dict(torch.load('model.pth'))
+model.eval()
+model = model.to('cuda')  # or 'mps', 'cpu'
+
+# Optional: compile for extra speed on CUDA
+model = torch.compile(model)
+
+# Load images
+rgb = cv2.imread('rgb.jpg')[:, :, ::-1]  # BGR to RGB
+depth = cv2.imread('depth.png', cv2.IMREAD_UNCHANGED) / 1000.0  # Convert to meters
+
+# Create similarity depth (inverse depth)
+simi_depth = np.zeros_like(depth)
+simi_depth[depth > 0] = 1 / depth[depth > 0]
+
+# Run inference with mixed precision
+with torch.amp.autocast('cuda', dtype=torch.float16):
+    pred_depth = model.infer_image(rgb, simi_depth, input_size=518)
+```
+
+## Model Training
+
+CDMs are trained on synthetic datasets generated using camera-specific noise models:
+
+1. **Noise Model Training**: Learn hole and value noise patterns from real camera data
+2. **Synthetic Data Generation**: Apply learned noise to clean simulation depth
+3. **CDM Training**: Train depth estimation model on synthetic noisy data
+
+Training datasets: HyperSim, DREDS, HISS, IRS (280,000+ images total)
+
+## Supported Cameras
+
+We currently provide pre-trained models available for:
+- Intel RealSense D405/D435/L515
+- Stereolabs ZED 2i (2 modes: Quality, Neural)
+- Microsoft Azure Kinect
+
+## File Structure
+
+```
+cdm/
+├── infer.py              # Main inference script
+├── setup.py              # Package installation
+├── rgbddepth/            # Core package
+│   ├── __init__.py
+│   ├── dpt.py            # Main RGBDDepth model
+│   ├── dinov2.py         # DINOv2 encoder
+│   ├── dinov2_layers/    # ViT transformer layers
+│   └── util/             # Utility functions
+│       ├── blocks.py     # Neural network blocks
+│       └── transform.py  # Image preprocessing
+└── README.md
+```
+
+## Performance
+
+### Accuracy
+
+This implementation achieves **pixel-perfect alignment** with the ByteDance reference:
+- ✅ **0 pixel difference** between vanilla and optimized inference (verified on test images)
+- ✅ **Identical checkpoint loading** (weights are fully compatible)
+- ✅ **Numerical precision preserved** (min=0.2036, max=1.1217, exact match)
+
+CDMs achieve state-of-the-art performance on metric depth estimation:
+- Superior accuracy compared to existing prompt-based depth models
+- Zero-shot generalization across different camera types
+- Real-time inference suitable for robot control (lightweight ViT variants)
+
+### Speed Benchmarks
+
+| Device | Mode | Precision | Time | vs Baseline | Notes |
+|--------|------|-----------|------|-------------|-------|
+| **CUDA** | Vanilla | FP32 | TBD | - | Reference |
+| **CUDA** | Optimized (xFormers) | FP32 | TBD | ~8% faster | Recommended |
+| **CUDA** | Optimized | FP16 | TBD | ~2× faster | Best speed |
+| **CUDA** | Optimized | BF16 | TBD | ~2× faster | Best stability |
+| **MPS** | Vanilla | FP32 | 1.34s | - | torch.compile: no gain |
+| **MPS** | Vanilla | FP16 | TBD | TBD | To be benchmarked |
+| **CPU** | Vanilla | FP32 | 13.37s | - | Optimizations: -11% slower |
+
+**Notes:**
+- **CUDA**: Optimizations auto-enabled by default (use `--no-optimize` to disable)
+- **MPS**: torch.compile provides no gain for Vision Transformers (~0% improvement)
+- **CPU**: torch.compile is counterproductive (compilation overhead > gains)
+- xFormers is CUDA-only (~8% faster than native SDPA)
+
+For detailed optimization strategies, see [OPTIMIZATION.md](OPTIMIZATION.md).
+
+## What's Different from Reference?
+
+This implementation maintains **100% compatibility** with ByteDance CDM while adding:
+
+### 1. Performance Optimizations
+- **xFormers support**: ~8% faster attention on CUDA (automatic fallback to SDPA)
+- **torch.compile**: JIT compilation (CUDA only, auto-enabled)
+- **Mixed precision**: FP16/BF16 support via `torch.amp.autocast`
+- **Device-specific strategies**: Optimizations only where beneficial
+
+### 2. Better CLI/API
+- `--device` flag: Force specific device (auto/cuda/mps/cpu)
+- `--precision` flag: Choose FP32/FP16/BF16
+- `--no-optimize` flag: Disable optimizations for debugging
+- Automatic device detection and optimization selection
+
+### 3. Improved Architecture
+- `FlexibleCrossAttention`: Inherits from `nn.MultiheadAttention` for checkpoint compatibility
+- Automatic backend selection: xFormers (CUDA) → SDPA (fallback)
+- Device-aware preprocessing: Uses model's device instead of auto-detection
+
+### 4. Code Quality
+- Type hints and better documentation
+- Cleaner argument parsing
+- Validation for precision/device combinations
+- Helpful warnings for incompatible configurations
+
+All changes are **backwards compatible** with original checkpoints and produce **identical numerical results**.
+
+## Citation
+
+If you use CDM in your research, please cite:
+
+```bibtex
+@article{liu2025manipulation,
+  title={Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots},
+  author={Liu, Minghuan and Zhu, Zhengbang and Han, Xiaoshen and Hu, Peng and Lin, Haotong and
+          Li, Xinyao and Chen, Jingxiao and Xu, Jiafeng and Yang, Yichu and Lin, Yunfeng and
+          Li, Xinghang and Yu, Yong and Zhang, Weinan and Kong, Tao and Kang, Bingyi},
+  journal={arXiv preprint},
+  year={2025}
+}
+```
+
+## License
+
+This project is licensed under the Apache 2.0 License. See [LICENSE](../LICENSE) for details.

app.py

@@ -0,0 +1,322 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
+# SPDX-License-Identifier: Apache-2.0
+
+"""Gradio demo for rgbd-depth on Hugging Face Spaces."""
+
+import gradio as gr
+import numpy as np
+import torch
+from PIL import Image
+
+from rgbddepth import RGBDDepth
+
+# Global model cache
+MODELS = {}
+
+
+def load_model(encoder: str, use_xformers: bool = False):
+    """Load model with caching."""
+    cache_key = f"{encoder}_{use_xformers}"
+
+    if cache_key not in MODELS:
+        # Model configs
+        configs = {
+            "vits": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384]},
+            "vitb": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768]},
+            "vitl": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024]},
+            "vitg": {"encoder": "vitg", "features": 384, "out_channels": [1536, 1536, 1536, 1536]},
+        }
+
+        config = configs[encoder].copy()
+        config["use_xformers"] = use_xformers
+
+        model = RGBDDepth(**config)
+
+        # Try to load weights if checkpoint exists
+        try:
+            checkpoint = torch.load(f"checkpoints/{encoder}.pt", map_location="cpu")
+            if "model" in checkpoint:
+                states = {k[7:]: v for k, v in checkpoint["model"].items()}
+            elif "state_dict" in checkpoint:
+                states = {k[9:]: v for k, v in checkpoint["state_dict"].items()}
+            else:
+                states = checkpoint
+
+            model.load_state_dict(states, strict=False)
+            print(f"✓ Loaded checkpoint for {encoder}")
+        except FileNotFoundError:
+            print(f"⚠ No checkpoint found for {encoder}, using random weights (demo only)")
+
+        # Move to GPU if available
+        device = "cuda" if torch.cuda.is_available() else "cpu"
+        model = model.to(device).eval()
+
+        MODELS[cache_key] = model
+
+    return MODELS[cache_key]
+
+
+def process_depth(
+    rgb_image: np.ndarray,
+    depth_image: np.ndarray,
+    encoder: str = "vitl",
+    input_size: int = 518,
+    depth_scale: float = 1000.0,
+    max_depth: float = 25.0,
+    use_xformers: bool = False,
+    precision: str = "fp32",
+    colormap: str = "Spectral",
+) -> tuple[Image.Image, str]:
+    """Process RGB-D depth refinement.
+
+    Args:
+        rgb_image: RGB image as numpy array [H, W, 3]
+        depth_image: Depth image as numpy array [H, W] or [H, W, 3]
+        encoder: Model encoder type
+        input_size: Input size for inference
+        depth_scale: Scale factor for depth values
+        max_depth: Maximum valid depth value
+        use_xformers: Whether to use xFormers (CUDA only)
+        precision: Precision mode (fp32/fp16/bf16)
+        colormap: Matplotlib colormap for visualization
+
+    Returns:
+        Tuple of (refined depth image, info message)
+    """
+    try:
+        # Validate inputs
+        if rgb_image is None:
+            return None, "❌ Please upload an RGB image"
+        if depth_image is None:
+            return None, "❌ Please upload a depth image"
+
+        # Convert depth to single channel if needed
+        if depth_image.ndim == 3:
+            depth_image = depth_image[:, :, 0]
+
+        # Normalize depth
+        depth_normalized = depth_image.astype(np.float32) / depth_scale
+        depth_normalized[depth_normalized > max_depth] = 0.0
+
+        # Create inverse depth (similarity depth)
+        simi_depth = np.zeros_like(depth_normalized)
+        valid_mask = depth_normalized > 0
+        simi_depth[valid_mask] = 1.0 / depth_normalized[valid_mask]
+
+        # Load model
+        model = load_model(encoder, use_xformers and torch.cuda.is_available())
+        device = next(model.parameters()).device
+
+        # Determine precision
+        if precision == "fp16" and device.type in ["cuda", "mps"]:
+            dtype = torch.float16
+        elif precision == "bf16" and device.type == "cuda":
+            dtype = torch.bfloat16
+        else:
+            dtype = None  # FP32
+
+        # Run inference
+        if dtype is not None:
+            device_type = "cuda" if device.type == "cuda" else "cpu"
+            with torch.amp.autocast(device_type=device_type, dtype=dtype):
+                pred = model.infer_image(rgb_image, simi_depth, input_size=input_size)
+        else:
+            pred = model.infer_image(rgb_image, simi_depth, input_size=input_size)
+
+        # Convert from inverse depth to depth
+        pred = np.where(pred > 1e-8, 1.0 / pred, 0.0)
+
+        # Colorize for visualization
+        try:
+            import matplotlib
+            import matplotlib.pyplot as plt
+
+            # Normalize to [0, 1]
+            pred_min, pred_max = pred.min(), pred.max()
+            if pred_max - pred_min > 1e-8:
+                pred_norm = (pred - pred_min) / (pred_max - pred_min)
+            else:
+                pred_norm = np.zeros_like(pred)
+
+            # Apply colormap
+            cm_func = matplotlib.colormaps[colormap]
+            pred_colored = cm_func(pred_norm, bytes=True)[:, :, :3]  # RGB only
+
+            # Create PIL Image
+            output_image = Image.fromarray(pred_colored)
+
+        except ImportError:
+            # Fallback to grayscale if matplotlib not available
+            pred_norm = ((pred - pred.min()) / (pred.max() - pred.min() + 1e-8) * 255).astype(np.uint8)
+            output_image = Image.fromarray(pred_norm, mode='L').convert('RGB')
+
+        # Create info message
+        info = f"""
+✅ **Refinement complete!**
+
+**Model:** {encoder.upper()}
+**Precision:** {precision.upper()}
+**Device:** {device.type.upper()}
+**Input size:** {input_size}px
+**Depth range:** {pred_min:.3f}m - {pred_max:.3f}m
+**xFormers:** {'✓ Enabled' if use_xformers and torch.cuda.is_available() else '✗ Disabled'}
+"""
+
+        return output_image, info.strip()
+
+    except Exception as e:
+        return None, f"❌ Error: {str(e)}"
+
+
+# Create Gradio interface
+with gr.Blocks(title="rgbd-depth Demo") as demo:
+    gr.Markdown("""
+    # 🎨 rgbd-depth: RGB-D Depth Refinement
+
+    High-quality depth map refinement using Vision Transformers. Based on [ByteDance's camera-depth-models](https://manipulation-as-in-simulation.github.io/).
+
+    ⚠️ **Note:** This demo uses random weights for demonstration. For real results:
+    1. Download checkpoints from [Hugging Face](https://huggingface.co/collections/depth-anything/camera-depth-models-68b521181dedd223f4b020db)
+    2. Place in `checkpoints/` directory
+    3. Restart the app
+    """)
+
+    with gr.Row():
+        with gr.Column():
+            gr.Markdown("### 📥 Inputs")
+
+            rgb_input = gr.Image(
+                label="RGB Image",
+                type="numpy",
+                height=300,
+            )
+
+            depth_input = gr.Image(
+                label="Input Depth Map",
+                type="numpy",
+                height=300,
+            )
+
+            with gr.Accordion("⚙️ Advanced Settings", open=False):
+                encoder_choice = gr.Radio(
+                    choices=["vits", "vitb", "vitl", "vitg"],
+                    value="vitl",
+                    label="Encoder Model",
+                    info="Larger = better quality but slower",
+                )
+
+                input_size = gr.Slider(
+                    minimum=256,
+                    maximum=1024,
+                    value=518,
+                    step=2,
+                    label="Input Size",
+                    info="Resolution for processing (higher = better but slower)",
+                )
+
+                depth_scale = gr.Number(
+                    value=1000.0,
+                    label="Depth Scale",
+                    info="Scale factor to convert depth values to meters",
+                )
+
+                max_depth = gr.Number(
+                    value=25.0,
+                    label="Max Depth (m)",
+                    info="Maximum valid depth value",
+                )
+
+                precision_choice = gr.Radio(
+                    choices=["fp32", "fp16", "bf16"],
+                    value="fp32",
+                    label="Precision",
+                    info="fp16/bf16 = faster but slightly less accurate (CUDA only)",
+                )
+
+                use_xformers = gr.Checkbox(
+                    value=False,
+                    label="Use xFormers (CUDA only)",
+                    info="~8% faster on CUDA with xFormers installed",
+                )
+
+                colormap_choice = gr.Dropdown(
+                    choices=["Spectral", "viridis", "plasma", "inferno", "magma", "turbo"],
+                    value="Spectral",
+                    label="Colormap",
+                    info="Visualization colormap",
+                )
+
+            process_btn = gr.Button("🚀 Refine Depth", variant="primary", size="lg")
+
+        with gr.Column():
+            gr.Markdown("### 📤 Output")
+
+            output_image = gr.Image(
+                label="Refined Depth Map",
+                type="pil",
+                height=600,
+            )
+
+            output_info = gr.Markdown()
+
+    # Example inputs
+    gr.Markdown("### 📸 Examples")
+    gr.Examples(
+        examples=[
+            ["example_data/color_12.png", "example_data/depth_12.png"],
+        ],
+        inputs=[rgb_input, depth_input],
+        label="Try with example images",
+    )
+
+    # Process button click
+    process_btn.click(
+        fn=process_depth,
+        inputs=[
+            rgb_input,
+            depth_input,
+            encoder_choice,
+            input_size,
+            depth_scale,
+            max_depth,
+            use_xformers,
+            precision_choice,
+            colormap_choice,
+        ],
+        outputs=[output_image, output_info],
+    )
+
+    # Footer
+    gr.Markdown("""
+    ---
+
+    ### 🔗 Links
+
+    - **GitHub:** [Aedelon/camera-depth-models](https://github.com/Aedelon/camera-depth-models)
+    - **PyPI:** [rgbd-depth](https://pypi.org/project/rgbd-depth/)
+    - **Paper:** [Manipulation-as-in-Simulation](https://manipulation-as-in-simulation.github.io/)
+
+    ### 📦 Install
+
+    ```bash
+    pip install rgbd-depth
+    ```
+
+    ### 💻 CLI Usage
+
+    ```bash
+    rgbd-depth \\
+      --model-path model.pt \\
+      --rgb-image input.jpg \\
+      --depth-image depth.png \\
+      --output refined.png
+    ```
+
+    ---
+
+    Built with ❤️ by [Aedelon](https://github.com/Aedelon) | Powered by [Gradio](https://gradio.app)
+    """)
+
+if __name__ == "__main__":
+    demo.launch()

requirements.txt

@@ -0,0 +1,16 @@
+# Hugging Face Spaces requirements
+# Generated from pyproject.toml - DO NOT install rgbd-depth itself (causes circular dependency)
+
+# Core dependencies (from pyproject.toml)
+torch>=2.0.0
+torchvision>=0.15.0
+opencv-python>=4.5.0
+numpy>=1.20.0
+Pillow>=9.0.0
+
+# Gradio demo
+gradio>=4.0.0
+matplotlib>=3.5.0
+
+# Model downloads from HuggingFace
+huggingface-hub>=0.16.0

rgbddepth/__init__.py

@@ -0,0 +1,12 @@
+"""RGBD Depth - Optimized RGB-D depth refinement using Vision Transformers.
+
+This package provides optimized depth refinement for RGB-D cameras with support
+for CUDA (xFormers), MPS (Apple Silicon), and CPU devices.
+"""
+
+__version__ = "1.0.2"
+
+from .dinov2 import DinoVisionTransformer
+from .dpt import RGBDDepth
+
+__all__ = ["RGBDDepth", "DinoVisionTransformer", "__version__"]

rgbddepth/dinov2.py

@@ -0,0 +1,441 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the Apache License, Version 2.0
+# found in the LICENSE file in the root directory of this source tree.
+
+# References:
+#   https://github.com/facebookresearch/dino/blob/main/vision_transformer.py
+#   https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer.py
+
+import logging
+import math
+from functools import partial
+from typing import Callable, Sequence, Tuple, Union
+
+import torch
+import torch.nn as nn
+import torch.utils.checkpoint
+from torch.nn.init import trunc_normal_
+
+from .dinov2_layers import MemEffAttention, Mlp
+from .dinov2_layers import NestedTensorBlock as Block
+from .dinov2_layers import PatchEmbed, SwiGLUFFNFused
+
+logger = logging.getLogger("dinov2")
+
+
+def named_apply(
+    fn: Callable, module: nn.Module, name="", depth_first=True, include_root=False
+) -> nn.Module:
+    if not depth_first and include_root:
+        fn(module=module, name=name)
+    for child_name, child_module in module.named_children():
+        child_name = ".".join((name, child_name)) if name else child_name
+        named_apply(
+            fn=fn,
+            module=child_module,
+            name=child_name,
+            depth_first=depth_first,
+            include_root=True,
+        )
+    if depth_first and include_root:
+        fn(module=module, name=name)
+    return module
+
+
+class BlockChunk(nn.ModuleList):
+    def forward(self, x):
+        for b in self:
+            x = b(x)
+        return x
+
+
+class DinoVisionTransformer(nn.Module):
+    def __init__(
+        self,
+        img_size=224,
+        patch_size=16,
+        in_chans=3,
+        embed_dim=768,
+        depth=12,
+        num_heads=12,
+        mlp_ratio=4.0,
+        qkv_bias=True,
+        ffn_bias=True,
+        proj_bias=True,
+        drop_path_rate=0.0,
+        drop_path_uniform=False,
+        init_values=None,  # for layerscale: None or 0 => no layerscale
+        embed_layer=PatchEmbed,
+        act_layer=nn.GELU,
+        block_fn=Block,
+        ffn_layer="mlp",
+        block_chunks=1,
+        num_register_tokens=0,
+        interpolate_antialias=False,
+        interpolate_offset=0.1,
+    ):
+        """
+        Args:
+            img_size (int, tuple): input image size
+            patch_size (int, tuple): patch size
+            in_chans (int): number of input channels
+            embed_dim (int): embedding dimension
+            depth (int): depth of transformer
+            num_heads (int): number of attention heads
+            mlp_ratio (int): ratio of mlp hidden dim to embedding dim
+            qkv_bias (bool): enable bias for qkv if True
+            proj_bias (bool): enable bias for proj in attn if True
+            ffn_bias (bool): enable bias for ffn if True
+            drop_path_rate (float): stochastic depth rate
+            drop_path_uniform (bool): apply uniform drop rate across blocks
+            weight_init (str): weight init scheme
+            init_values (float): layer-scale init values
+            embed_layer (nn.Module): patch embedding layer
+            act_layer (nn.Module): MLP activation layer
+            block_fn (nn.Module): transformer block class
+            ffn_layer (str): "mlp", "swiglu", "swiglufused" or "identity"
+            block_chunks: (int) split block sequence into block_chunks units for FSDP wrap
+            num_register_tokens: (int) number of extra cls tokens (so-called "registers")
+            interpolate_antialias: (str) flag to apply anti-aliasing when interpolating positional embeddings
+            interpolate_offset: (float) work-around offset to apply when interpolating positional embeddings
+        """
+        super().__init__()
+        norm_layer = partial(nn.LayerNorm, eps=1e-6)
+
+        self.num_features = self.embed_dim = (
+            embed_dim  # num_features for consistency with other models
+        )
+        self.num_tokens = 1
+        self.n_blocks = depth
+        self.num_heads = num_heads
+        self.patch_size = patch_size
+        self.num_register_tokens = num_register_tokens
+        self.interpolate_antialias = interpolate_antialias
+        self.interpolate_offset = interpolate_offset
+
+        self.patch_embed = embed_layer(
+            img_size=img_size,
+            patch_size=patch_size,
+            in_chans=in_chans,
+            embed_dim=embed_dim,
+        )
+        num_patches = self.patch_embed.num_patches
+
+        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
+        self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + self.num_tokens, embed_dim))
+        assert num_register_tokens >= 0
+        self.register_tokens = (
+            nn.Parameter(torch.zeros(1, num_register_tokens, embed_dim))
+            if num_register_tokens
+            else None
+        )
+
+        if drop_path_uniform is True:
+            dpr = [drop_path_rate] * depth
+        else:
+            dpr = [
+                x.item() for x in torch.linspace(0, drop_path_rate, depth)
+            ]  # stochastic depth decay rule
+
+        if ffn_layer == "mlp":
+            logger.info("using MLP layer as FFN")
+            ffn_layer = Mlp
+        elif ffn_layer == "swiglufused" or ffn_layer == "swiglu":
+            logger.info("using SwiGLU layer as FFN")
+            ffn_layer = SwiGLUFFNFused
+        elif ffn_layer == "identity":
+            logger.info("using Identity layer as FFN")
+
+            def f(*args, **kwargs):
+                return nn.Identity()
+
+            ffn_layer = f
+        else:
+            raise NotImplementedError
+
+        blocks_list = [
+            block_fn(
+                dim=embed_dim,
+                num_heads=num_heads,
+                mlp_ratio=mlp_ratio,
+                qkv_bias=qkv_bias,
+                proj_bias=proj_bias,
+                ffn_bias=ffn_bias,
+                drop_path=dpr[i],
+                norm_layer=norm_layer,
+                act_layer=act_layer,
+                ffn_layer=ffn_layer,
+                init_values=init_values,
+            )
+            for i in range(depth)
+        ]
+        if block_chunks > 0:
+            self.chunked_blocks = True
+            chunked_blocks = []
+            chunksize = depth // block_chunks
+            for i in range(0, depth, chunksize):
+                # this is to keep the block index consistent if we chunk the block list
+                chunked_blocks.append([nn.Identity()] * i + blocks_list[i : i + chunksize])
+            self.blocks = nn.ModuleList([BlockChunk(p) for p in chunked_blocks])
+        else:
+            self.chunked_blocks = False
+            self.blocks = nn.ModuleList(blocks_list)
+
+        self.norm = norm_layer(embed_dim)
+        self.head = nn.Identity()
+
+        self.mask_token = nn.Parameter(torch.zeros(1, embed_dim))
+
+        self.init_weights()
+
+    def init_weights(self):
+        trunc_normal_(self.pos_embed, std=0.02)
+        nn.init.normal_(self.cls_token, std=1e-6)
+        if self.register_tokens is not None:
+            nn.init.normal_(self.register_tokens, std=1e-6)
+        named_apply(init_weights_vit_timm, self)
+
+    def interpolate_pos_encoding(self, x, w, h):
+        previous_dtype = x.dtype
+        npatch = x.shape[1] - 1
+        N = self.pos_embed.shape[1] - 1
+        if npatch == N and w == h:
+            return self.pos_embed
+        pos_embed = self.pos_embed.float()
+        class_pos_embed = pos_embed[:, 0]
+        patch_pos_embed = pos_embed[:, 1:]
+        dim = x.shape[-1]
+        w0 = w // self.patch_size
+        h0 = h // self.patch_size
+        # we add a small number to avoid floating point error in the interpolation
+        # see discussion at https://github.com/facebookresearch/dino/issues/8
+        # DINOv2 with register modify the interpolate_offset from 0.1 to 0.0
+        w0, h0 = w0 + self.interpolate_offset, h0 + self.interpolate_offset
+        # w0, h0 = w0 + 0.1, h0 + 0.1
+
+        sqrt_N = math.sqrt(N)
+        sx, sy = float(w0) / sqrt_N, float(h0) / sqrt_N
+        patch_pos_embed = nn.functional.interpolate(
+            patch_pos_embed.reshape(1, int(sqrt_N), int(sqrt_N), dim).permute(0, 3, 1, 2),
+            scale_factor=(sx, sy),
+            # (int(w0), int(h0)), # to solve the upsampling shape issue
+            mode="bicubic",
+            antialias=self.interpolate_antialias,
+        )
+
+        assert int(w0) == patch_pos_embed.shape[-2]
+        assert int(h0) == patch_pos_embed.shape[-1]
+        patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim)
+        return torch.cat((class_pos_embed.unsqueeze(0), patch_pos_embed), dim=1).to(previous_dtype)
+
+    def prepare_tokens_with_masks(self, x, masks=None):
+        B, nc, w, h = x.shape
+        x = self.patch_embed(x)
+        if masks is not None:
+            x = torch.where(masks.unsqueeze(-1), self.mask_token.to(x.dtype).unsqueeze(0), x)
+
+        x = torch.cat((self.cls_token.expand(x.shape[0], -1, -1), x), dim=1)
+        x = x + self.interpolate_pos_encoding(x, w, h)
+
+        if self.register_tokens is not None:
+            x = torch.cat(
+                (
+                    x[:, :1],
+                    self.register_tokens.expand(x.shape[0], -1, -1),
+                    x[:, 1:],
+                ),
+                dim=1,
+            )
+
+        return x
+
+    def forward_features_list(self, x_list, masks_list):
+        x = [self.prepare_tokens_with_masks(x, masks) for x, masks in zip(x_list, masks_list)]
+        for blk in self.blocks:
+            x = blk(x)
+
+        all_x = x
+        output = []
+        for x, masks in zip(all_x, masks_list):
+            x_norm = self.norm(x)
+            output.append(
+                {
+                    "x_norm_clstoken": x_norm[:, 0],
+                    "x_norm_regtokens": x_norm[:, 1 : self.num_register_tokens + 1],
+                    "x_norm_patchtokens": x_norm[:, self.num_register_tokens + 1 :],
+                    "x_prenorm": x,
+                    "masks": masks,
+                }
+            )
+        return output
+
+    def forward_features(self, x, masks=None):
+        if isinstance(x, list):
+            return self.forward_features_list(x, masks)
+
+        x = self.prepare_tokens_with_masks(x, masks)
+
+        for blk in self.blocks:
+            x = blk(x)
+
+        x_norm = self.norm(x)
+        return {
+            "x_norm_clstoken": x_norm[:, 0],
+            "x_norm_regtokens": x_norm[:, 1 : self.num_register_tokens + 1],
+            "x_norm_patchtokens": x_norm[:, self.num_register_tokens + 1 :],
+            "x_prenorm": x,
+            "masks": masks,
+        }
+
+    def _get_intermediate_layers_not_chunked(self, x, n=1):
+        x = self.prepare_tokens_with_masks(x)
+        # If n is an int, take the n last blocks. If it's a list, take them
+        output, total_block_len = [], len(self.blocks)
+        blocks_to_take = range(total_block_len - n, total_block_len) if isinstance(n, int) else n
+        for i, blk in enumerate(self.blocks):
+            x = blk(x)
+            if i in blocks_to_take:
+                output.append(x)
+        assert len(output) == len(
+            blocks_to_take
+        ), f"only {len(output)} / {len(blocks_to_take)} blocks found"
+        return output
+
+    def _get_intermediate_layers_chunked(self, x, n=1):
+        x = self.prepare_tokens_with_masks(x)
+        output, i, total_block_len = [], 0, len(self.blocks[-1])
+        # If n is an int, take the n last blocks. If it's a list, take them
+        blocks_to_take = range(total_block_len - n, total_block_len) if isinstance(n, int) else n
+        for block_chunk in self.blocks:
+            for blk in block_chunk[i:]:  # Passing the nn.Identity()
+                x = blk(x)
+                if i in blocks_to_take:
+                    output.append(x)
+                i += 1
+        assert len(output) == len(
+            blocks_to_take
+        ), f"only {len(output)} / {len(blocks_to_take)} blocks found"
+        return output
+
+    def get_intermediate_layers(
+        self,
+        x: torch.Tensor,
+        n: Union[int, Sequence] = 1,  # Layers or n last layers to take
+        reshape: bool = False,
+        return_class_token: bool = False,
+        norm=True,
+    ) -> Tuple[Union[torch.Tensor, Tuple[torch.Tensor]]]:
+        if self.chunked_blocks:
+            outputs = self._get_intermediate_layers_chunked(x, n)
+        else:
+            outputs = self._get_intermediate_layers_not_chunked(x, n)
+        if norm:
+            outputs = [self.norm(out) for out in outputs]
+        class_tokens = [out[:, 0] for out in outputs]
+        outputs = [out[:, 1 + self.num_register_tokens :] for out in outputs]
+        if reshape:
+            B, _, w, h = x.shape
+            outputs = [
+                out.reshape(B, w // self.patch_size, h // self.patch_size, -1)
+                .permute(0, 3, 1, 2)
+                .contiguous()
+                for out in outputs
+            ]
+        if return_class_token:
+            return tuple(zip(outputs, class_tokens))
+        return tuple(outputs)
+
+    def forward(self, *args, is_training=False, **kwargs):
+        ret = self.forward_features(*args, **kwargs)
+        if is_training:
+            return ret
+        else:
+            return self.head(ret["x_norm_clstoken"])
+
+
+def init_weights_vit_timm(module: nn.Module, name: str = ""):
+    """ViT weight initialization, original timm impl (for reproducibility)"""
+    if isinstance(module, nn.Linear):
+        trunc_normal_(module.weight, std=0.02)
+        if module.bias is not None:
+            nn.init.zeros_(module.bias)
+
+
+def vit_small(patch_size=16, num_register_tokens=0, **kwargs):
+    model = DinoVisionTransformer(
+        patch_size=patch_size,
+        embed_dim=384,
+        depth=12,
+        num_heads=6,
+        mlp_ratio=4,
+        block_fn=partial(Block, attn_class=MemEffAttention),
+        num_register_tokens=num_register_tokens,
+        **kwargs,
+    )
+    return model
+
+
+def vit_base(patch_size=16, num_register_tokens=0, **kwargs):
+    model = DinoVisionTransformer(
+        patch_size=patch_size,
+        embed_dim=768,
+        depth=12,
+        num_heads=12,
+        mlp_ratio=4,
+        block_fn=partial(Block, attn_class=MemEffAttention),
+        num_register_tokens=num_register_tokens,
+        **kwargs,
+    )
+    return model
+
+
+def vit_large(patch_size=16, num_register_tokens=0, **kwargs):
+    model = DinoVisionTransformer(
+        patch_size=patch_size,
+        embed_dim=1024,
+        depth=24,
+        num_heads=16,
+        mlp_ratio=4,
+        block_fn=partial(Block, attn_class=MemEffAttention),
+        num_register_tokens=num_register_tokens,
+        **kwargs,
+    )
+    return model
+
+
+def vit_giant2(patch_size=16, num_register_tokens=0, **kwargs):
+    """
+    Close to ViT-giant, with embed-dim 1536 and 24 heads => embed-dim per head 64
+    """
+    model = DinoVisionTransformer(
+        patch_size=patch_size,
+        embed_dim=1536,
+        depth=40,
+        num_heads=24,
+        mlp_ratio=4,
+        block_fn=partial(Block, attn_class=MemEffAttention),
+        num_register_tokens=num_register_tokens,
+        **kwargs,
+    )
+    return model
+
+
+def DINOv2(model_name):
+    model_zoo = {
+        "vits": vit_small,
+        "vitb": vit_base,
+        "vitl": vit_large,
+        "vitg": vit_giant2,
+    }
+
+    return model_zoo[model_name](
+        img_size=518,
+        patch_size=14,
+        init_values=1.0,
+        ffn_layer="mlp" if model_name != "vitg" else "swiglufused",
+        block_chunks=0,
+        num_register_tokens=0,
+        interpolate_antialias=False,
+        interpolate_offset=0.1,
+    )

rgbddepth/dinov2_layers/__init__.py

@@ -0,0 +1,20 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+from .attention import MemEffAttention
+from .block import NestedTensorBlock
+from .mlp import Mlp
+from .patch_embed import PatchEmbed
+from .swiglu_ffn import SwiGLUFFN, SwiGLUFFNFused
+
+__all__ = [
+    "MemEffAttention",
+    "NestedTensorBlock",
+    "Mlp",
+    "PatchEmbed",
+    "SwiGLUFFN",
+    "SwiGLUFFNFused",
+]

rgbddepth/dinov2_layers/attention.py

@@ -0,0 +1,81 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+#   https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+#   https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer.py
+
+import logging
+
+from torch import Tensor, nn
+
+logger = logging.getLogger("dinov2")
+
+
+try:
+    from xformers.ops import memory_efficient_attention, unbind
+
+    XFORMERS_AVAILABLE = True
+except ImportError:
+    logger.warning("xFormers not available")
+    XFORMERS_AVAILABLE = False
+
+
+class Attention(nn.Module):
+    def __init__(
+        self,
+        dim: int,
+        num_heads: int = 8,
+        qkv_bias: bool = False,
+        proj_bias: bool = True,
+        attn_drop: float = 0.0,
+        proj_drop: float = 0.0,
+    ) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        head_dim = dim // num_heads
+        self.scale = head_dim**-0.5
+
+        self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
+        self.attn_drop = nn.Dropout(attn_drop)
+        self.proj = nn.Linear(dim, dim, bias=proj_bias)
+        self.proj_drop = nn.Dropout(proj_drop)
+
+    def forward(self, x: Tensor) -> Tensor:
+        B, N, C = x.shape
+        qkv = (
+            self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
+        )
+
+        q, k, v = qkv[0] * self.scale, qkv[1], qkv[2]
+        attn = q @ k.transpose(-2, -1)
+
+        attn = attn.softmax(dim=-1)
+        attn = self.attn_drop(attn)
+
+        x = (attn @ v).transpose(1, 2).reshape(B, N, C)
+        x = self.proj(x)
+        x = self.proj_drop(x)
+        return x
+
+
+class MemEffAttention(Attention):
+    def forward(self, x: Tensor, attn_bias=None) -> Tensor:
+        if not XFORMERS_AVAILABLE:
+            assert attn_bias is None, "xFormers is required for nested tensors usage"
+            return super().forward(x)
+
+        B, N, C = x.shape
+        qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads)
+
+        q, k, v = unbind(qkv, 2)
+
+        x = memory_efficient_attention(q, k, v, attn_bias=attn_bias)
+        x = x.reshape([B, N, C])
+
+        x = self.proj(x)
+        x = self.proj_drop(x)
+        return x

rgbddepth/dinov2_layers/block.py

@@ -0,0 +1,266 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+#   https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+#   https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/patch_embed.py
+
+import logging
+from typing import Any, Callable, Dict, List, Tuple
+
+import torch
+from torch import Tensor, nn
+
+from .attention import Attention, MemEffAttention
+from .drop_path import DropPath
+from .layer_scale import LayerScale
+from .mlp import Mlp
+
+logger = logging.getLogger("dinov2")
+
+
+try:
+    from xformers.ops import fmha, index_select_cat, scaled_index_add
+
+    XFORMERS_AVAILABLE = True
+except ImportError:
+    logger.warning("xFormers not available")
+    XFORMERS_AVAILABLE = False
+
+
+class Block(nn.Module):
+    def __init__(
+        self,
+        dim: int,
+        num_heads: int,
+        mlp_ratio: float = 4.0,
+        qkv_bias: bool = False,
+        proj_bias: bool = True,
+        ffn_bias: bool = True,
+        drop: float = 0.0,
+        attn_drop: float = 0.0,
+        init_values=None,
+        drop_path: float = 0.0,
+        act_layer: Callable[..., nn.Module] = nn.GELU,
+        norm_layer: Callable[..., nn.Module] = nn.LayerNorm,
+        attn_class: Callable[..., nn.Module] = Attention,
+        ffn_layer: Callable[..., nn.Module] = Mlp,
+    ) -> None:
+        super().__init__()
+        # print(f"biases: qkv: {qkv_bias}, proj: {proj_bias}, ffn: {ffn_bias}")
+        self.norm1 = norm_layer(dim)
+        self.attn = attn_class(
+            dim,
+            num_heads=num_heads,
+            qkv_bias=qkv_bias,
+            proj_bias=proj_bias,
+            attn_drop=attn_drop,
+            proj_drop=drop,
+        )
+        self.ls1 = LayerScale(dim, init_values=init_values) if init_values else nn.Identity()
+        self.drop_path1 = DropPath(drop_path) if drop_path > 0.0 else nn.Identity()
+
+        self.norm2 = norm_layer(dim)
+        mlp_hidden_dim = int(dim * mlp_ratio)
+        self.mlp = ffn_layer(
+            in_features=dim,
+            hidden_features=mlp_hidden_dim,
+            act_layer=act_layer,
+            drop=drop,
+            bias=ffn_bias,
+        )
+        self.ls2 = LayerScale(dim, init_values=init_values) if init_values else nn.Identity()
+        self.drop_path2 = DropPath(drop_path) if drop_path > 0.0 else nn.Identity()
+
+        self.sample_drop_ratio = drop_path
+
+    def forward(self, x: Tensor) -> Tensor:
+        def attn_residual_func(x: Tensor) -> Tensor:
+            return self.ls1(self.attn(self.norm1(x)))
+
+        def ffn_residual_func(x: Tensor) -> Tensor:
+            return self.ls2(self.mlp(self.norm2(x)))
+
+        if self.training and self.sample_drop_ratio > 0.1:
+            # the overhead is compensated only for a drop path rate larger than 0.1
+            x = drop_add_residual_stochastic_depth(
+                x,
+                residual_func=attn_residual_func,
+                sample_drop_ratio=self.sample_drop_ratio,
+            )
+            x = drop_add_residual_stochastic_depth(
+                x,
+                residual_func=ffn_residual_func,
+                sample_drop_ratio=self.sample_drop_ratio,
+            )
+        elif self.training and self.sample_drop_ratio > 0.0:
+            x = x + self.drop_path1(attn_residual_func(x))
+            x = x + self.drop_path1(ffn_residual_func(x))  # FIXME: drop_path2
+        else:
+            x = x + attn_residual_func(x)
+            x = x + ffn_residual_func(x)
+        return x
+
+
+def drop_add_residual_stochastic_depth(
+    x: Tensor,
+    residual_func: Callable[[Tensor], Tensor],
+    sample_drop_ratio: float = 0.0,
+) -> Tensor:
+    # 1) extract subset using permutation
+    b, n, d = x.shape
+    sample_subset_size = max(int(b * (1 - sample_drop_ratio)), 1)
+    brange = (torch.randperm(b, device=x.device))[:sample_subset_size]
+    x_subset = x[brange]
+
+    # 2) apply residual_func to get residual
+    residual = residual_func(x_subset)
+
+    x_flat = x.flatten(1)
+    residual = residual.flatten(1)
+
+    residual_scale_factor = b / sample_subset_size
+
+    # 3) add the residual
+    x_plus_residual = torch.index_add(
+        x_flat, 0, brange, residual.to(dtype=x.dtype), alpha=residual_scale_factor
+    )
+    return x_plus_residual.view_as(x)
+
+
+def get_branges_scales(x, sample_drop_ratio=0.0):
+    b, n, d = x.shape
+    sample_subset_size = max(int(b * (1 - sample_drop_ratio)), 1)
+    brange = (torch.randperm(b, device=x.device))[:sample_subset_size]
+    residual_scale_factor = b / sample_subset_size
+    return brange, residual_scale_factor
+
+
+def add_residual(x, brange, residual, residual_scale_factor, scaling_vector=None):
+    if scaling_vector is None:
+        x_flat = x.flatten(1)
+        residual = residual.flatten(1)
+        x_plus_residual = torch.index_add(
+            x_flat, 0, brange, residual.to(dtype=x.dtype), alpha=residual_scale_factor
+        )
+    else:
+        x_plus_residual = scaled_index_add(
+            x,
+            brange,
+            residual.to(dtype=x.dtype),
+            scaling=scaling_vector,
+            alpha=residual_scale_factor,
+        )
+    return x_plus_residual
+
+
+attn_bias_cache: Dict[Tuple, Any] = {}
+
+
+def get_attn_bias_and_cat(x_list, branges=None):
+    """
+    this will perform the index select, cat the tensors, and provide the attn_bias from cache
+    """
+    batch_sizes = (
+        [b.shape[0] for b in branges] if branges is not None else [x.shape[0] for x in x_list]
+    )
+    all_shapes = tuple((b, x.shape[1]) for b, x in zip(batch_sizes, x_list))
+    if all_shapes not in attn_bias_cache.keys():
+        seqlens = []
+        for b, x in zip(batch_sizes, x_list):
+            for _ in range(b):
+                seqlens.append(x.shape[1])
+        attn_bias = fmha.BlockDiagonalMask.from_seqlens(seqlens)
+        attn_bias._batch_sizes = batch_sizes
+        attn_bias_cache[all_shapes] = attn_bias
+
+    if branges is not None:
+        cat_tensors = index_select_cat([x.flatten(1) for x in x_list], branges).view(
+            1, -1, x_list[0].shape[-1]
+        )
+    else:
+        tensors_bs1 = tuple(x.reshape([1, -1, *x.shape[2:]]) for x in x_list)
+        cat_tensors = torch.cat(tensors_bs1, dim=1)
+
+    return attn_bias_cache[all_shapes], cat_tensors
+
+
+def drop_add_residual_stochastic_depth_list(
+    x_list: List[Tensor],
+    residual_func: Callable[[Tensor, Any], Tensor],
+    sample_drop_ratio: float = 0.0,
+    scaling_vector=None,
+) -> Tensor:
+    # 1) generate random set of indices for dropping samples in the batch
+    branges_scales = [get_branges_scales(x, sample_drop_ratio=sample_drop_ratio) for x in x_list]
+    branges = [s[0] for s in branges_scales]
+    residual_scale_factors = [s[1] for s in branges_scales]
+
+    # 2) get attention bias and index+concat the tensors
+    attn_bias, x_cat = get_attn_bias_and_cat(x_list, branges)
+
+    # 3) apply residual_func to get residual, and split the result
+    residual_list = attn_bias.split(residual_func(x_cat, attn_bias=attn_bias))  # type: ignore
+
+    outputs = []
+    for x, brange, residual, residual_scale_factor in zip(
+        x_list, branges, residual_list, residual_scale_factors
+    ):
+        outputs.append(
+            add_residual(x, brange, residual, residual_scale_factor, scaling_vector).view_as(x)
+        )
+    return outputs
+
+
+class NestedTensorBlock(Block):
+    def forward_nested(self, x_list: List[Tensor]) -> List[Tensor]:
+        """
+        x_list contains a list of tensors to nest together and run
+        """
+        assert isinstance(self.attn, MemEffAttention)
+
+        if self.training and self.sample_drop_ratio > 0.0:
+
+            def attn_residual_func(x: Tensor, attn_bias=None) -> Tensor:
+                return self.attn(self.norm1(x), attn_bias=attn_bias)
+
+            def ffn_residual_func(x: Tensor, attn_bias=None) -> Tensor:
+                return self.mlp(self.norm2(x))
+
+            x_list = drop_add_residual_stochastic_depth_list(
+                x_list,
+                residual_func=attn_residual_func,
+                sample_drop_ratio=self.sample_drop_ratio,
+                scaling_vector=(self.ls1.gamma if isinstance(self.ls1, LayerScale) else None),
+            )
+            x_list = drop_add_residual_stochastic_depth_list(
+                x_list,
+                residual_func=ffn_residual_func,
+                sample_drop_ratio=self.sample_drop_ratio,
+                scaling_vector=(self.ls2.gamma if isinstance(self.ls1, LayerScale) else None),
+            )
+            return x_list
+        else:
+
+            def attn_residual_func(x: Tensor, attn_bias=None) -> Tensor:
+                return self.ls1(self.attn(self.norm1(x), attn_bias=attn_bias))
+
+            def ffn_residual_func(x: Tensor, attn_bias=None) -> Tensor:
+                return self.ls2(self.mlp(self.norm2(x)))
+
+            attn_bias, x = get_attn_bias_and_cat(x_list)
+            x = x + attn_residual_func(x, attn_bias=attn_bias)
+            x = x + ffn_residual_func(x)
+            return attn_bias.split(x)
+
+    def forward(self, x_or_x_list):
+        if isinstance(x_or_x_list, Tensor):
+            return super().forward(x_or_x_list)
+        elif isinstance(x_or_x_list, list):
+            assert XFORMERS_AVAILABLE, "Please install xFormers for nested tensors usage"
+            return self.forward_nested(x_or_x_list)
+        else:
+            raise AssertionError

rgbddepth/dinov2_layers/drop_path.py

@@ -0,0 +1,35 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+#   https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+#   https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/drop.py
+
+
+from torch import nn
+
+
+def drop_path(x, drop_prob: float = 0.0, training: bool = False):
+    if drop_prob == 0.0 or not training:
+        return x
+    keep_prob = 1 - drop_prob
+    shape = (x.shape[0],) + (1,) * (x.ndim - 1)  # work with diff dim tensors, not just 2D ConvNets
+    random_tensor = x.new_empty(shape).bernoulli_(keep_prob)
+    if keep_prob > 0.0:
+        random_tensor.div_(keep_prob)
+    output = x * random_tensor
+    return output
+
+
+class DropPath(nn.Module):
+    """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks)."""
+
+    def __init__(self, drop_prob=None):
+        super(DropPath, self).__init__()
+        self.drop_prob = drop_prob
+
+    def forward(self, x):
+        return drop_path(x, self.drop_prob, self.training)

rgbddepth/dinov2_layers/layer_scale.py

@@ -0,0 +1,27 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Modified from: https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/vision_transformer.py#L103-L110
+
+from typing import Union
+
+import torch
+from torch import Tensor, nn
+
+
+class LayerScale(nn.Module):
+    def __init__(
+        self,
+        dim: int,
+        init_values: Union[float, Tensor] = 1e-5,
+        inplace: bool = False,
+    ) -> None:
+        super().__init__()
+        self.inplace = inplace
+        self.gamma = nn.Parameter(init_values * torch.ones(dim))
+
+    def forward(self, x: Tensor) -> Tensor:
+        return x.mul_(self.gamma) if self.inplace else x * self.gamma

rgbddepth/dinov2_layers/mlp.py

@@ -0,0 +1,41 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+#   https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+#   https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/mlp.py
+
+
+from typing import Callable, Optional
+
+from torch import Tensor, nn
+
+
+class Mlp(nn.Module):
+    def __init__(
+        self,
+        in_features: int,
+        hidden_features: Optional[int] = None,
+        out_features: Optional[int] = None,
+        act_layer: Callable[..., nn.Module] = nn.GELU,
+        drop: float = 0.0,
+        bias: bool = True,
+    ) -> None:
+        super().__init__()
+        out_features = out_features or in_features
+        hidden_features = hidden_features or in_features
+        self.fc1 = nn.Linear(in_features, hidden_features, bias=bias)
+        self.act = act_layer()
+        self.fc2 = nn.Linear(hidden_features, out_features, bias=bias)
+        self.drop = nn.Dropout(drop)
+
+    def forward(self, x: Tensor) -> Tensor:
+        x = self.fc1(x)
+        x = self.act(x)
+        x = self.drop(x)
+        x = self.fc2(x)
+        x = self.drop(x)
+        return x

rgbddepth/dinov2_layers/patch_embed.py

@@ -0,0 +1,93 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+#   https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+#   https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/patch_embed.py
+
+from typing import Callable, Optional, Tuple, Union
+
+import torch.nn as nn
+from torch import Tensor
+
+
+def make_2tuple(x):
+    if isinstance(x, tuple):
+        assert len(x) == 2
+        return x
+
+    assert isinstance(x, int)
+    return (x, x)
+
+
+class PatchEmbed(nn.Module):
+    """
+    2D image to patch embedding: (B,C,H,W) -> (B,N,D)
+
+    Args:
+        img_size: Image size.
+        patch_size: Patch token size.
+        in_chans: Number of input image channels.
+        embed_dim: Number of linear projection output channels.
+        norm_layer: Normalization layer.
+    """
+
+    def __init__(
+        self,
+        img_size: Union[int, Tuple[int, int]] = 224,
+        patch_size: Union[int, Tuple[int, int]] = 16,
+        in_chans: int = 3,
+        embed_dim: int = 768,
+        norm_layer: Optional[Callable] = None,
+        flatten_embedding: bool = True,
+    ) -> None:
+        super().__init__()
+
+        image_HW = make_2tuple(img_size)
+        patch_HW = make_2tuple(patch_size)
+        patch_grid_size = (
+            image_HW[0] // patch_HW[0],
+            image_HW[1] // patch_HW[1],
+        )
+
+        self.img_size = image_HW
+        self.patch_size = patch_HW
+        self.patches_resolution = patch_grid_size
+        self.num_patches = patch_grid_size[0] * patch_grid_size[1]
+
+        self.in_chans = in_chans
+        self.embed_dim = embed_dim
+
+        self.flatten_embedding = flatten_embedding
+
+        self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_HW, stride=patch_HW)
+        self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()
+
+    def forward(self, x: Tensor) -> Tensor:
+        _, _, H, W = x.shape
+        patch_H, patch_W = self.patch_size
+
+        assert (
+            H % patch_H == 0
+        ), f"Input image height {H} is not a multiple of patch height {patch_H}"
+        assert (
+            W % patch_W == 0
+        ), f"Input image width {W} is not a multiple of patch width: {patch_W}"
+
+        x = self.proj(x)  # B C H W
+        H, W = x.size(2), x.size(3)
+        x = x.flatten(2).transpose(1, 2)  # B HW C
+        x = self.norm(x)
+        if not self.flatten_embedding:
+            x = x.reshape(-1, H, W, self.embed_dim)  # B H W C
+        return x
+
+    def flops(self) -> float:
+        Ho, Wo = self.patches_resolution
+        flops = Ho * Wo * self.embed_dim * self.in_chans * (self.patch_size[0] * self.patch_size[1])
+        if self.norm is not None:
+            flops += Ho * Wo * self.embed_dim
+        return flops

rgbddepth/dinov2_layers/swiglu_ffn.py

@@ -0,0 +1,63 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import Callable, Optional
+
+import torch.nn.functional as F
+from torch import Tensor, nn
+
+
+class SwiGLUFFN(nn.Module):
+    def __init__(
+        self,
+        in_features: int,
+        hidden_features: Optional[int] = None,
+        out_features: Optional[int] = None,
+        act_layer: Callable[..., nn.Module] = None,
+        drop: float = 0.0,
+        bias: bool = True,
+    ) -> None:
+        super().__init__()
+        out_features = out_features or in_features
+        hidden_features = hidden_features or in_features
+        self.w12 = nn.Linear(in_features, 2 * hidden_features, bias=bias)
+        self.w3 = nn.Linear(hidden_features, out_features, bias=bias)
+
+    def forward(self, x: Tensor) -> Tensor:
+        x12 = self.w12(x)
+        x1, x2 = x12.chunk(2, dim=-1)
+        hidden = F.silu(x1) * x2
+        return self.w3(hidden)
+
+
+try:
+    from xformers.ops import SwiGLU
+
+    XFORMERS_AVAILABLE = True
+except ImportError:
+    SwiGLU = SwiGLUFFN
+    XFORMERS_AVAILABLE = False
+
+
+class SwiGLUFFNFused(SwiGLU):
+    def __init__(
+        self,
+        in_features: int,
+        hidden_features: Optional[int] = None,
+        out_features: Optional[int] = None,
+        act_layer: Callable[..., nn.Module] = None,
+        drop: float = 0.0,
+        bias: bool = True,
+    ) -> None:
+        out_features = out_features or in_features
+        hidden_features = hidden_features or in_features
+        hidden_features = (int(hidden_features * 2 / 3) + 7) // 8 * 8
+        super().__init__(
+            in_features=in_features,
+            hidden_features=hidden_features,
+            out_features=out_features,
+            bias=bias,
+        )

rgbddepth/dpt.py

@@ -0,0 +1,312 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
+# SPDX-License-Identifier: Apache-2.0
+
+import cv2
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torchvision.transforms import Compose
+
+from .dinov2 import DINOv2
+from .flexible_attention import FlexibleCrossAttention
+from .util.blocks import FeatureFusionBlock, _make_scratch
+from .util.transform import NormalizeImage, PrepareForNet, Resize
+
+
+def _make_fusion_block(features, use_bn, size=None):
+    return FeatureFusionBlock(
+        features,
+        nn.ReLU(False),
+        deconv=False,
+        bn=use_bn,
+        expand=False,
+        align_corners=True,
+        size=size,
+    )
+
+
+class ConvBlock(nn.Module):
+    def __init__(self, in_feature, out_feature):
+        super().__init__()
+
+        self.conv_block = nn.Sequential(
+            nn.Conv2d(in_feature, out_feature, kernel_size=3, stride=1, padding=1),
+            nn.BatchNorm2d(out_feature),
+            nn.ReLU(True),
+        )
+
+    def forward(self, x):
+        return self.conv_block(x)
+
+
+class DPTHead(nn.Module):
+    def __init__(
+        self,
+        in_channels,
+        features=256,
+        use_bn=False,
+        out_channels=[256, 512, 1024, 1024],
+        use_clstoken=False,
+        sigact_out=False,
+    ):
+        super(DPTHead, self).__init__()
+
+        self.use_clstoken = use_clstoken
+
+        self.projects = nn.ModuleList(
+            [
+                nn.Conv2d(
+                    in_channels=in_channels,
+                    out_channels=out_channel,
+                    kernel_size=1,
+                    stride=1,
+                    padding=0,
+                )
+                for out_channel in out_channels
+            ]
+        )
+
+        self.resize_layers = nn.ModuleList(
+            [
+                nn.ConvTranspose2d(
+                    in_channels=out_channels[0],
+                    out_channels=out_channels[0],
+                    kernel_size=4,
+                    stride=4,
+                    padding=0,
+                ),
+                nn.ConvTranspose2d(
+                    in_channels=out_channels[1],
+                    out_channels=out_channels[1],
+                    kernel_size=2,
+                    stride=2,
+                    padding=0,
+                ),
+                nn.Identity(),
+                nn.Conv2d(
+                    in_channels=out_channels[3],
+                    out_channels=out_channels[3],
+                    kernel_size=3,
+                    stride=2,
+                    padding=1,
+                ),
+            ]
+        )
+
+        if use_clstoken:
+            self.readout_projects = nn.ModuleList()
+            for _ in range(len(self.projects)):
+                self.readout_projects.append(
+                    nn.Sequential(nn.Linear(2 * in_channels, in_channels), nn.GELU())
+                )
+
+        self.scratch = _make_scratch(
+            out_channels,
+            features,
+            groups=1,
+            expand=False,
+        )
+
+        self.scratch.stem_transpose = None
+
+        self.scratch.refinenet1 = _make_fusion_block(features, use_bn)
+        self.scratch.refinenet2 = _make_fusion_block(features, use_bn)
+        self.scratch.refinenet3 = _make_fusion_block(features, use_bn)
+        self.scratch.refinenet4 = _make_fusion_block(features, use_bn)
+
+        head_features_1 = features
+        head_features_2 = 32
+
+        self.scratch.output_conv1 = nn.Conv2d(
+            head_features_1, head_features_1 // 2, kernel_size=3, stride=1, padding=1
+        )
+
+        if not sigact_out:
+            self.scratch.output_conv2 = nn.Sequential(
+                nn.Conv2d(
+                    head_features_1 // 2,
+                    head_features_2,
+                    kernel_size=3,
+                    stride=1,
+                    padding=1,
+                ),
+                nn.ReLU(True),
+                nn.Conv2d(head_features_2, 1, kernel_size=1, stride=1, padding=0),
+                nn.ReLU(True),
+                nn.Identity(),
+            )
+        else:
+            self.scratch.output_conv2 = nn.Sequential(
+                nn.Conv2d(
+                    head_features_1 // 2,
+                    head_features_2,
+                    kernel_size=3,
+                    stride=1,
+                    padding=1,
+                ),
+                nn.ReLU(True),
+                nn.Conv2d(head_features_2, 1, kernel_size=1, stride=1, padding=0),
+                nn.Sigmoid(),
+            )
+
+    def forward(self, out_features, patch_h, patch_w):
+        out = []
+        for i, x in enumerate(out_features):
+            if self.use_clstoken:
+                x, cls_token = x[0], x[1]
+                readout = cls_token.unsqueeze(1).expand_as(x)
+                x = self.readout_projects[i](torch.cat((x, readout), -1))
+            else:
+                x = x[0]
+
+            x = x.permute(0, 2, 1).reshape((x.shape[0], x.shape[-1], patch_h, patch_w))
+
+            x = self.projects[i](x)
+            x = self.resize_layers[i](x)
+
+            out.append(x)
+
+        layer_1, layer_2, layer_3, layer_4 = out
+
+        layer_1_rn = self.scratch.layer1_rn(layer_1)
+        layer_2_rn = self.scratch.layer2_rn(layer_2)
+        layer_3_rn = self.scratch.layer3_rn(layer_3)
+        layer_4_rn = self.scratch.layer4_rn(layer_4)
+
+        path_4 = self.scratch.refinenet4(layer_4_rn, size=layer_3_rn.shape[2:])
+        path_3 = self.scratch.refinenet3(path_4, layer_3_rn, size=layer_2_rn.shape[2:])
+        path_2 = self.scratch.refinenet2(path_3, layer_2_rn, size=layer_1_rn.shape[2:])
+        path_1 = self.scratch.refinenet1(path_2, layer_1_rn)
+
+        out = self.scratch.output_conv1(path_1)
+        out = F.interpolate(
+            out,
+            (int(patch_h * 14), int(patch_w * 14)),
+            mode="bilinear",
+            align_corners=True,
+        )
+        out = self.scratch.output_conv2(out)
+
+        return out
+
+
+class RGBDDepth(nn.Module):
+    def __init__(
+        self,
+        encoder="vitl",
+        features=256,
+        out_channels=[256, 512, 1024, 1024],
+        use_bn=False,
+        use_clstoken=False,
+        max_depth=20.0,
+        use_xformers=False,
+    ):
+        super(RGBDDepth, self).__init__()
+
+        self.intermediate_layer_idx = {
+            "vits": [2, 5, 8, 11],
+            "vitb": [2, 5, 8, 11],
+            "vitl": [4, 11, 17, 23],
+            "vitg": [9, 19, 29, 39],
+        }
+
+        self.max_depth = max_depth
+
+        self.encoder = encoder
+        self.pretrained = DINOv2(model_name=encoder)
+        self.depth_pretrained = DINOv2(model_name=encoder)
+
+        # self.depth_head = DPTHead(self.pretrained.embed_dim, features, use_bn, out_channels=out_channels, use_clstoken=use_clstoken, sigact_out=False)
+        self.depth_head_rgbd = DPTHead(
+            self.pretrained.embed_dim * 2,
+            features,
+            use_bn,
+            out_channels=out_channels,
+            use_clstoken=use_clstoken,
+            sigact_out=False,
+        )
+
+        # cross attention with xFormers support
+        num_heads = 4
+        self.crossAtts = nn.ModuleList(
+            [
+                FlexibleCrossAttention(
+                    self.pretrained.embed_dim, num_heads, use_xformers=use_xformers
+                )
+                for _ in range(4)
+            ]
+        )
+
+    def forward(self, x):
+        rgb, depth = x[:, :3], x[:, 3:]
+        patch_h, patch_w = x.shape[-2] // 14, x.shape[-1] // 14
+
+        with torch.no_grad():
+            features_rgb = self.pretrained.get_intermediate_layers(
+                rgb, self.intermediate_layer_idx[self.encoder], return_class_token=True
+            )
+
+        features_depth = self.depth_pretrained.get_intermediate_layers(
+            depth.repeat(1, 3, 1, 1),
+            self.intermediate_layer_idx[self.encoder],
+            return_class_token=True,
+        )
+        features = []
+        for f_rgb, f_depth, crossAtt in zip(features_rgb, features_depth, self.crossAtts):
+            B, N, C = f_rgb[0].shape
+            tf_rgb = f_rgb[0].reshape(B * N, 1, C)
+            tf_depth = f_depth[0].reshape(B * N, 1, C)
+            token_feat = torch.concat((tf_rgb, tf_depth), axis=1)
+            att_feat, _ = crossAtt(token_feat, token_feat, token_feat)
+            att_feat = att_feat.reshape(B * N, 2, C).sum(axis=1).reshape(B, N, C)
+
+            feat = torch.concat((f_rgb[0], att_feat), axis=2)
+            cls_t = torch.concat((f_rgb[1], f_depth[1]), axis=1)
+            tuples = (feat, cls_t)
+            features.append(tuples)
+        depth = self.depth_head_rgbd(features, patch_h, patch_w)
+        depth = F.relu(depth)
+        return depth.squeeze(1)
+
+    @torch.no_grad()
+    def infer_image(self, raw_image, depth_low_res, input_size=518):
+        inputs, (h, w) = self.image2tensor(raw_image, depth_low_res, input_size)
+        pred_depth = self.forward(inputs)
+        pred_depth = F.interpolate(pred_depth[:, None], (h, w), mode="nearest")[0, 0]
+        return pred_depth.cpu().numpy()
+
+    def image2tensor(self, raw_image, depth, input_size=518):
+        transform = Compose(
+            [
+                Resize(
+                    width=input_size,
+                    height=input_size,
+                    resize_target=True,
+                    keep_aspect_ratio=True,
+                    ensure_multiple_of=14,
+                    resize_method="lower_bound",
+                    image_interpolation_method=cv2.INTER_CUBIC,
+                ),
+                NormalizeImage(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+                PrepareForNet(),
+            ]
+        )
+
+        h, w = raw_image.shape[:2]
+
+        image = cv2.cvtColor(raw_image, cv2.COLOR_BGR2RGB) / 255.0
+        prepared = transform({"image": image, "depth": depth})
+        image = prepared["image"]
+        image = torch.from_numpy(image).unsqueeze(0)
+
+        depth = prepared["depth"]
+        depth = torch.from_numpy(depth).unsqueeze(0).unsqueeze(0)
+
+        inputs = torch.cat((image, depth), dim=1)
+
+        # Use the same device as model parameters
+        device = next(self.parameters()).device
+        inputs = inputs.to(device)
+
+        return inputs, (h, w)

rgbddepth/flexible_attention.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
+# SPDX-License-Identifier: Apache-2.0
+
+"""Flexible cross-attention module with xFormers support and automatic fallback."""
+
+import torch
+import torch.nn as nn
+
+
+class FlexibleCrossAttention(nn.MultiheadAttention):
+    """Cross-attention with optional xFormers support and automatic fallback to SDPA.
+
+    This module inherits from nn.MultiheadAttention to ensure weight compatibility.
+    It overrides forward() to use xFormers when available and requested.
+
+    Uses:
+    1. xFormers memory-efficient attention (CUDA only, if installed and use_xformers=True)
+    2. PyTorch native SDPA (Scaled Dot Product Attention, PyTorch 2.0+, default)
+    3. Standard MultiheadAttention (fallback for older PyTorch versions)
+
+    Args:
+        embed_dim: Total dimension of the model
+        num_heads: Number of parallel attention heads
+        use_xformers: Whether to attempt using xFormers (only works on CUDA)
+    """
+
+    def __init__(self, embed_dim, num_heads, use_xformers=False, **kwargs):
+        # Initialize parent with batch_first=True to match original usage
+        super().__init__(embed_dim, num_heads, batch_first=True, **kwargs)
+
+        self.embed_dim = embed_dim
+        self.num_heads = num_heads
+        self.head_dim = embed_dim // num_heads
+
+        # Check if xFormers is available and requested
+        self.use_xformers = use_xformers and self._check_xformers()
+
+    def _check_xformers(self):
+        """Check if xFormers is available for import.
+
+        Returns:
+            bool: True if xFormers can be imported, False otherwise
+        """
+        try:
+            import importlib.util
+
+            return importlib.util.find_spec("xformers.ops") is not None
+        except (ImportError, ValueError):
+            return False
+
+    def forward(self, query, key, value, **kwargs):
+        """Forward pass with automatic backend selection.
+
+        Args:
+            query: Query tensor of shape [B, N, C]
+            key: Key tensor of shape [B, N, C]
+            value: Value tensor of shape [B, N, C]
+
+        Returns:
+            tuple: (output, attention_weights)
+                - output: Attention output of shape [B, N, C]
+                - attention_weights: None (not computed for efficiency)
+        """
+        if not self.use_xformers:
+            # Standard path using parent nn.MultiheadAttention (with SDPA in PyTorch 2.0+)
+            # This uses the original weights (in_proj_weight, out_proj) from checkpoint
+            return super().forward(query, key, value, need_weights=False, **kwargs)
+        else:
+            # xFormers memory-efficient attention path
+            import xformers.ops as xops
+
+            # Use parent's projection weights for Q, K, V
+            # in_proj_weight contains concatenated [W_q; W_k; W_v]
+            # This ensures we use the exact same weights as standard MultiheadAttention
+            if self.in_proj_weight is not None:
+                # Split the combined in_proj_weight into Q, K, V weights
+                w_q, w_k, w_v = self.in_proj_weight.chunk(3, dim=0)
+                b_q, b_k, b_v = None, None, None
+                if self.in_proj_bias is not None:
+                    b_q, b_k, b_v = self.in_proj_bias.chunk(3, dim=0)
+
+                # Apply projections using the same weights as standard attention
+                q = torch.nn.functional.linear(query, w_q, b_q)
+                k = torch.nn.functional.linear(key, w_k, b_k)
+                v = torch.nn.functional.linear(value, w_v, b_v)
+            else:
+                # Separate projection weights (shouldn't happen with default config)
+                q = torch.nn.functional.linear(query, self.q_proj_weight, self.in_proj_bias)
+                k = torch.nn.functional.linear(key, self.k_proj_weight)
+                v = torch.nn.functional.linear(value, self.v_proj_weight)
+
+            # Reshape for multi-head attention: [B, N, C] -> [B, N, H, C//H]
+            B, N, C = q.shape
+            q = q.reshape(B, N, self.num_heads, self.head_dim)
+            k = k.reshape(B, N, self.num_heads, self.head_dim)
+            v = v.reshape(B, N, self.num_heads, self.head_dim)
+
+            # Apply xFormers memory-efficient attention
+            # This is significantly faster and uses less memory than standard attention
+            out = xops.memory_efficient_attention(q, k, v)
+
+            # Reshape back: [B, N, H, C//H] -> [B, N, C]
+            out = out.reshape(B, N, C)
+
+            # Use parent's output projection (same weights as standard attention)
+            out = torch.nn.functional.linear(out, self.out_proj.weight, self.out_proj.bias)
+
+            return out, None

rgbddepth/util/blocks.py

@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
+# SPDX-License-Identifier: Apache-2.0
+
+import torch.nn as nn
+
+
+def _make_scratch(in_shape, out_shape, groups=1, expand=False):
+    scratch = nn.Module()
+
+    out_shape1 = out_shape
+    out_shape2 = out_shape
+    out_shape3 = out_shape
+    if len(in_shape) >= 4:
+        out_shape4 = out_shape
+
+    if expand:
+        out_shape1 = out_shape
+        out_shape2 = out_shape * 2
+        out_shape3 = out_shape * 4
+        if len(in_shape) >= 4:
+            out_shape4 = out_shape * 8
+
+    scratch.layer1_rn = nn.Conv2d(
+        in_shape[0],
+        out_shape1,
+        kernel_size=3,
+        stride=1,
+        padding=1,
+        bias=False,
+        groups=groups,
+    )
+    scratch.layer2_rn = nn.Conv2d(
+        in_shape[1],
+        out_shape2,
+        kernel_size=3,
+        stride=1,
+        padding=1,
+        bias=False,
+        groups=groups,
+    )
+    scratch.layer3_rn = nn.Conv2d(
+        in_shape[2],
+        out_shape3,
+        kernel_size=3,
+        stride=1,
+        padding=1,
+        bias=False,
+        groups=groups,
+    )
+    if len(in_shape) >= 4:
+        scratch.layer4_rn = nn.Conv2d(
+            in_shape[3],
+            out_shape4,
+            kernel_size=3,
+            stride=1,
+            padding=1,
+            bias=False,
+            groups=groups,
+        )
+
+    return scratch
+
+
+class ResidualConvUnit(nn.Module):
+    """Residual convolution module."""
+
+    def __init__(self, features, activation, bn):
+        """Init.
+
+        Args:
+            features (int): number of features
+        """
+        super().__init__()
+
+        self.bn = bn
+
+        self.groups = 1
+
+        self.conv1 = nn.Conv2d(
+            features,
+            features,
+            kernel_size=3,
+            stride=1,
+            padding=1,
+            bias=True,
+            groups=self.groups,
+        )
+
+        self.conv2 = nn.Conv2d(
+            features,
+            features,
+            kernel_size=3,
+            stride=1,
+            padding=1,
+            bias=True,
+            groups=self.groups,
+        )
+
+        if self.bn:
+            self.bn1 = nn.BatchNorm2d(features)
+            self.bn2 = nn.BatchNorm2d(features)
+
+        self.activation = activation
+
+        self.skip_add = nn.quantized.FloatFunctional()
+
+    def forward(self, x):
+        """Forward pass.
+
+        Args:
+            x (tensor): input
+
+        Returns:
+            tensor: output
+        """
+
+        out = self.activation(x)
+        out = self.conv1(out)
+        if self.bn:
+            out = self.bn1(out)
+
+        out = self.activation(out)
+        out = self.conv2(out)
+        if self.bn:
+            out = self.bn2(out)
+
+        if self.groups > 1:
+            out = self.conv_merge(out)
+
+        return self.skip_add.add(out, x)
+
+
+class FeatureFusionBlock(nn.Module):
+    """Feature fusion block."""
+
+    def __init__(
+        self,
+        features,
+        activation,
+        deconv=False,
+        bn=False,
+        expand=False,
+        align_corners=True,
+        size=None,
+    ):
+        """Init.
+
+        Args:
+            features (int): number of features
+        """
+        super(FeatureFusionBlock, self).__init__()
+
+        self.deconv = deconv
+        self.align_corners = align_corners
+
+        self.groups = 1
+
+        self.expand = expand
+        out_features = features
+        if self.expand:
+            out_features = features // 2
+
+        self.out_conv = nn.Conv2d(
+            features,
+            out_features,
+            kernel_size=1,
+            stride=1,
+            padding=0,
+            bias=True,
+            groups=1,
+        )
+
+        self.resConfUnit1 = ResidualConvUnit(features, activation, bn)
+        self.resConfUnit2 = ResidualConvUnit(features, activation, bn)
+
+        self.skip_add = nn.quantized.FloatFunctional()
+
+        self.size = size
+
+    def forward(self, *xs, size=None):
+        """Forward pass.
+
+        Returns:
+            tensor: output
+        """
+        output = xs[0]
+
+        if len(xs) == 2:
+            res = self.resConfUnit1(xs[1])
+            output = self.skip_add.add(output, res)
+
+        output = self.resConfUnit2(output)
+
+        if (size is None) and (self.size is None):
+            modifier = {"scale_factor": 2}
+        elif size is None:
+            modifier = {"size": self.size}
+        else:
+            modifier = {"size": size}
+
+        output = nn.functional.interpolate(
+            output, **modifier, mode="bilinear", align_corners=self.align_corners
+        )
+
+        output = self.out_conv(output)
+
+        return output

rgbddepth/util/transform.py

@@ -0,0 +1,169 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
+# SPDX-License-Identifier: Apache-2.0
+
+import cv2
+import numpy as np
+
+
+class Resize(object):
+    """Resize sample to given size (width, height)."""
+
+    def __init__(
+        self,
+        width,
+        height,
+        resize_target=True,
+        keep_aspect_ratio=False,
+        ensure_multiple_of=1,
+        resize_method="lower_bound",
+        image_interpolation_method=cv2.INTER_AREA,
+    ):
+        """Init.
+
+        Args:
+            width (int): desired output width
+            height (int): desired output height
+            resize_target (bool, optional):
+                True: Resize the full sample (image, mask, target).
+                False: Resize image only.
+                Defaults to True.
+            keep_aspect_ratio (bool, optional):
+                True: Keep the aspect ratio of the input sample.
+                Output sample might not have the given width and height, and
+                resize behaviour depends on the parameter 'resize_method'.
+                Defaults to False.
+            ensure_multiple_of (int, optional):
+                Output width and height is constrained to be multiple of this parameter.
+                Defaults to 1.
+            resize_method (str, optional):
+                "lower_bound": Output will be at least as large as the given size.
+                "upper_bound": Output will be at max as large as the given size. (Output size might be smaller than given size.)
+                "minimal": Scale as least as possible.  (Output size might be smaller than given size.)
+                Defaults to "lower_bound".
+        """
+        self.__width = width
+        self.__height = height
+
+        self.__resize_target = resize_target
+        self.__keep_aspect_ratio = keep_aspect_ratio
+        self.__multiple_of = ensure_multiple_of
+        self.__resize_method = resize_method
+        self.__image_interpolation_method = image_interpolation_method
+
+    def constrain_to_multiple_of(self, x, min_val=0, max_val=None):
+        y = (np.round(x / self.__multiple_of) * self.__multiple_of).astype(int)
+
+        if max_val is not None and y > max_val:
+            y = (np.floor(x / self.__multiple_of) * self.__multiple_of).astype(int)
+
+        if y < min_val:
+            y = (np.ceil(x / self.__multiple_of) * self.__multiple_of).astype(int)
+
+        return y
+
+    def get_size(self, width, height):
+        # determine new height and width
+        scale_height = self.__height / height
+        scale_width = self.__width / width
+
+        if self.__keep_aspect_ratio:
+            if self.__resize_method == "lower_bound":
+                # scale such that output size is lower bound
+                if scale_width > scale_height:
+                    # fit width
+                    scale_height = scale_width
+                else:
+                    # fit height
+                    scale_width = scale_height
+            elif self.__resize_method == "upper_bound":
+                # scale such that output size is upper bound
+                if scale_width < scale_height:
+                    # fit width
+                    scale_height = scale_width
+                else:
+                    # fit height
+                    scale_width = scale_height
+            elif self.__resize_method == "minimal":
+                # scale as least as possbile
+                if abs(1 - scale_width) < abs(1 - scale_height):
+                    # fit width
+                    scale_height = scale_width
+                else:
+                    # fit height
+                    scale_width = scale_height
+            else:
+                raise ValueError(f"resize_method {self.__resize_method} not implemented")
+
+        if self.__resize_method == "lower_bound":
+            new_height = self.constrain_to_multiple_of(scale_height * height, min_val=self.__height)
+            new_width = self.constrain_to_multiple_of(scale_width * width, min_val=self.__width)
+        elif self.__resize_method == "upper_bound":
+            new_height = self.constrain_to_multiple_of(scale_height * height, max_val=self.__height)
+            new_width = self.constrain_to_multiple_of(scale_width * width, max_val=self.__width)
+        elif self.__resize_method == "minimal":
+            new_height = self.constrain_to_multiple_of(scale_height * height)
+            new_width = self.constrain_to_multiple_of(scale_width * width)
+        else:
+            raise ValueError(f"resize_method {self.__resize_method} not implemented")
+
+        return (new_width, new_height)
+
+    def __call__(self, sample):
+        width, height = self.get_size(sample["image"].shape[1], sample["image"].shape[0])
+
+        # resize sample
+        sample["image"] = cv2.resize(
+            sample["image"],
+            (width, height),
+            interpolation=self.__image_interpolation_method,
+        )
+
+        if self.__resize_target:
+            if "depth" in sample:
+                sample["depth"] = cv2.resize(
+                    sample["depth"], (width, height), interpolation=cv2.INTER_NEAREST
+                )
+
+            if "mask" in sample:
+                sample["mask"] = cv2.resize(
+                    sample["mask"].astype(np.float32),
+                    (width, height),
+                    interpolation=cv2.INTER_NEAREST,
+                )
+
+        return sample
+
+
+class NormalizeImage(object):
+    """Normlize image by given mean and std."""
+
+    def __init__(self, mean, std):
+        self.__mean = mean
+        self.__std = std
+
+    def __call__(self, sample):
+        sample["image"] = (sample["image"] - self.__mean) / self.__std
+
+        return sample
+
+
+class PrepareForNet(object):
+    """Prepare sample for usage as network input."""
+
+    def __init__(self):
+        pass
+
+    def __call__(self, sample):
+        image = np.transpose(sample["image"], (2, 0, 1))
+        sample["image"] = np.ascontiguousarray(image).astype(np.float32)
+
+        if "depth" in sample:
+            depth = sample["depth"].astype(np.float32)
+            sample["depth"] = np.ascontiguousarray(depth)
+
+        if "mask" in sample:
+            sample["mask"] = sample["mask"].astype(np.float32)
+            sample["mask"] = np.ascontiguousarray(sample["mask"])
+
+        return sample