Post-doctoral researcher; Hybrid Implicit Neural Representations for Next-Generation Learned Video Compression

JoinInterDigital'sVideo Lab to pioneer the next generation of energy-efficient AI for video. As a postdoctoral researcher, you will help shape technologies that reduce the environmental footprint of generative AI while contributing to international standards and publishingcutting-edgeresearch.

About InterDigital

InterDigital is a global research and development company focused primarily on wireless, video, artificial intelligence (“AI”), and related technologies. We design and develop foundational technologies that enable connected, immersive experiences in a broad range of communications and entertainment products and services. We license our innovations worldwide to companies providing such products and services, including makers of wireless communications devices, consumer electronics, IoT devices, cars and other motor vehicles, and providers of cloud-based services such as video streaming. As a leader in wireless technology, our engineers have designed and developed a wide range of innovations that are used in wireless products and networks, from the earliest digital cellular systems to 5G and today’s most advanced Wi-Fi technologies. We are also a leader in video processing and video encoding/decoding technology, with a significant AI research effort that intersects with both wireless and video technologies. Founded in 1972, InterDigital is listed on Nasdaq.

InterDigital is a registered trademark of InterDigital, Inc.

For more information, visit: www.interdigital.com.

Background and Motivation

The continuous growth of video traffic generated by streaming platforms, immersive media, cloud gaming, autonomous systems, and machine vision is placing unprecedented pressure on communication and storage infrastructures. Although conventional video codecs such as H.266/VVC, AV1 and HEVC continue to improve coding efficiency through decades of algorithmic refinements, their block-based architecture is approaching diminishing returns. Their design relies on handcrafted signal processing modules-including prediction, transforms, quantization, in-loop filtering and entropy coding-that have been carefully optimized but remain fundamentally constrained by assumptions about natural image statistics.

Learned Video Compression (LVC) has emerged during the past decade as a promising alternative. Instead of optimizing individual codec modules independently, learned codecs formulate compression as a global end-to-end optimization problem. Neural networks jointly learn the analysis transform, synthesis transform, entropy model and motion representation by minimizing a rate-distortion objectivewhere (D) measures reconstruction distortion while (R) estimates the coding rate through differentiable entropy models.

Recent learned image and video codecs have demonstrated compression performances comparable to-or surpassing-the latest standard codecs for several operating points. More importantly, they provide a flexible framework in which every component can be jointly optimized for a target application.

Despite these impressive advances, several important challenges remain before learned video codecs can become a practical replacement for conventional standards.

First, current architectures remain computationally demanding. Deep convolutional encoders, decoders and optical-flow estimation networks require millions of parameters and substantial computational resources during both training and inference.

Second, entropy modeling remains one of the principal bottlenecks. State-of-the-art autoregressive models improve coding efficiency but introduce strong sequential dependencies that significantly reduce decoding parallelism.

Third, current codecs still represent images and videos through discrete latent feature maps. While highly effective, these representations inherit many limitations of grid-based signals, including resolution dependence, interpolation artifacts and limited geometric adaptability.

Finally, temporal redundancy is often modeled explicitly through optical flow or recurrent networks, both of which become increasingly expensive for long sequences or high-resolution content.

These limitations motivate the exploration of alternative signal representations capable of providing richer spatial and temporal modeling while maintaining efficient entropy coding.

Hybrid Implicit Neural Representations

Implicit Neural Representations (INRs) have recently emerged as a fundamentally different paradigm for representing visual signals. Rather than storing an image or a video as discrete samples on a grid, an INR represents the signal as a continuous functionwhere the neural network maps spatial (or spatio-temporal) coordinates

XΧ directly to signal valuesΥΥ.

Continuous representations naturally provide:

• arbitrary-resolution reconstruction,

• smooth interpolation,

• compact parameterization,

• geometric continuity.

However, pure coordinate-based INRs exhibit several limitations for compression. They often require relatively large neural networks, suffer from slow convergence, and struggle to reproduce highly textured regions without substantially increasing model complexity.

Hybrid Implicit Neural Representations (Hybrid INRs) have recently addressed these limitations by combining learned latent grids with coordinate-based neural decoders.

Instead of relying solely on coordinates, the decoder receives both

• local latent features extracted from hierarchical latent grids,

• continuous spatial coordinates.

The latent grids capture high-frequency local structures while the implicit decoder reconstructs the continuous signal. Hierarchical latent resolutions allow coarse image structures to be represented at low spatial resolutions and fine details at higher resolutions.

Among recent examples, Cool-Chic[Ladune et al., 2003]and C3[Kim et al., 2024]havedemonstrated that hybrid INRs constitute an extremely competitive framework for learned image compression. Its hierarchical latent representation, lightweight implicit decoder and learned entropy model achieve state-of-the-art rate-distortion performance while requiring significantly fewer decoder parameters than conventional CNN-based architectures.

The advantages of hybrid INRs include:

• continuous image representation,

• compact decoder architectures,

• scalable latent hierarchies,

• compatibility with autoregressive entropy models,

• excellent adaptation to image geometry.

Nevertheless, current hybrid INR methods remain largely restricted to still images. Extending this framework to video introduces new scientific questions regarding temporal representation, temporal prediction and entropy modeling[Leguay et al., 2024].

Extending Hybrid INRs to Learned Video Compression

The central objective of this project is to develop a new generation of learned video codecs based on spatio-temporal hybrid implicit neural representations.Instead of treating each frame independently, the proposed framework will model an entire video sequence as a continuous function where the decoder jointly exploits spatial coordinates, temporal coordinates and hierarchical latent representations.The key challenge lies in introducing temporal redundancy into the latent hierarchy without sacrificing the compactness that makes hybrid INRs attractive.Several complementary research directions will be investigated.

Hierarchical Spatio-Temporal Latent Representation

• Current hybrid INR codecs organize latent grids across multiple spatial resolutions. This project proposes extending the representationto the temporal dimension.This could includetemporal latent pyramids,key-frame and inter-frame latent decomposition,multi-scale temporal feature grids,adaptive temporal resolutions depending on motion complexity…

• Continuous Temporal Prediction

• Conventional learned video codecs typically estimate optical flow between successive frames.Hybrid INRs provide an opportunity to replace explicit motion estimation by continuous temporal modeling, such aslatent-space temporal prediction,learned temporal embeddings…

• Hierarchical Context Modeling

• Entropy coding remains one of the largest contributors to coding performance.Inspired by recent advances in hierarchical latent modeling for image compression, previously decoded latent levels and previous temporal instants can provide rich contextual information for entropy prediction.

• Lightweight Neural Architectures

One of the strengths of hybrid INRs is the possibility of using extremely compact decoders.The projectaims to continueinvestigatinglightweight architecturesaiming to reduceboth computational complexity and memory footprint while preserving coding performance.

Expected Scientific Contributions

The proposed research will contribute to the development of a unified framework for learned video compression based onimplicit neural representation.

Expected contributions include:

• novel spatio-temporal hybrid INR architectures for video compression;

• lightweight decoder architectures suitable for practical deployment;

• comprehensive rate-distortion-complexity evaluation against state-of-the-art learned video codecs and conventional standards such as VVC.

Impact

This project addresses several major challenges facing next-generation multimedia compression: coding efficiency, computational complexity, scalability and representation flexibility. By extending hybrid implicit neural representations from images to videos, it seeks to establish a new paradigm in which spatial and temporal information are jointly modeled through continuous neural functions and hierarchical latent representations.

Such a framework has the potential to bridge the gap between neural signal representations and practical video coding systems, contributing both to the theoretical understanding of continuous neural compression and to the development of future AI-native video codecs suitable for real-world deployment.

Bibliography

[Kim et al., 2024]Kim, Hyunjik, et al."C3: High-performance and low-complexity neural compression from a single image or video." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

[Ladune et al., 2003]Ladune, Théo, et al."Cool-chic: Coordinate-based low complexity hierarchical image codec." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[Leguay et al., 2024]Leguay, Thomas, et al. "Cool-chic video: Learned video coding with 800 parameters." 2024 Data compression conference (DCC). IEEE, 2024.

Location: Rennes, France

InterDigital is an equal employment opportunity employer. InterDigital will not engage in or tolerate unlawful discrimination with regard to any employment decision, policy or practice based on a person’s sex, gender, pregnancy (including childbirth, breastfeeding and related medical conditions), age, race, color, religion, creed, national origin, ancestry, citizenship, military status, veteran status, mental or physical disability, medical condition, genetic information, sexual orientation, gender identity or expression, or any other factor protected by applicable federal, state or local law. This policy applies to all terms and conditions of employment, including, but not limited to, recruiting, hiring, compensation, benefits, training, assignments, evaluations, coaching, promotion, discipline, discharge and layoff.