Video Player & Timestamps

Overview

The Multimodal Video RAG system uses specialized frontend components to synchronize search results with video playback. Because the system indexes both visual frames and audio transcripts, the player components are designed to handle precise timestamp seeking, allowing users to jump directly to the relevant "moment" identified by the AI.

Core Components

VideoPlayer

The VideoPlayer is the foundational UI component used throughout the application to render video content. It is a controlled component that supports direct URL playback and basic configuration.

Usage:

import { VideoPlayer } from '@/components/common/VideoPlayer';

export default function MyComponent() {
    return (
        <VideoPlayer 
            url="https://www.youtube.com/watch?v=..." 
            autoPlay={false} 
        />
    );
}

Properties:

| Property | Type | Description | | :--- | :--- | :--- | | url | string | The source URL of the video (e.g., YouTube). | | autoPlay | boolean | (Optional) Whether the video should start playing immediately on load. |

VideoPlayerWrapper

The VideoPlayerWrapper is a context-aware component designed for RAG workflows. Instead of requiring a direct URL, it accepts a video_id and a start timestamp. This component is used when displaying successful ingestion results or navigating from search hits.

Usage:

import { VideoPlayerWrapper } from '@/components/search/VideoPlayerWrapper';

function SearchResultDetail({ videoId, startTime }) {
    return (
        <VideoPlayerWrapper 
            video_id={videoId} 
            start={startTime} 
        />
    );
}

Properties:

| Property | Type | Description | | :--- | :--- | :--- | | video_id | string | The internal unique identifier for the ingested video. | | start | number | The timestamp in seconds where the player should begin or seek to. |

Timestamp Integration

The RAG system returns timestamps within several API responses. These values are used to drive the player components.

Search Results

When performing a semantic search, the SearchResult object provides precise start and end points for content matches.

interface SearchResult {
    video_id: string;
    timestamp_start: number; // Start of the relevant segment in seconds
    timestamp_end: number;   // End of the relevant segment in seconds
    content: string;         // The transcript text or visual description
    chunk_type: 'audio' | 'visual';
}

Chat Sources

The Chat Assistant provides ChatSource objects that link AI-generated answers back to specific moments in the video library.

interface ChatSource {
    video_id: string;
    timestamp_seconds: number; // The exact point of reference
    transcript?: string;       // Text context (if audio match)
    visual_context?: string;   // Visual description (if frame match)
}

Practical Implementation: Seeking to a Result

To implement a "Jump to Timestamp" feature, the application maps the timestamp_start from a search result to the start prop of the VideoPlayerWrapper.

1. User performs a search: The backend returns a list of SearchResult objects.
2. User clicks a result: The application updates the state with the video_id and timestamp_start.
3. Player updates: The VideoPlayerWrapper detects the new timestamp and utilizes the underlying player API to seek the video to the specified second.

This flow ensures that when the AI identifies a specific event (e.g., "Elephants appear at 9.0s"), the user can verify the result with a single click.