Video upscaling in browser video players using AMD FidelityFX or similar technology

[tonebacas]

Would be nice to be able to use image upscaling on video players, as an optional feature you could toggle for any individual video player.

[Jon]

Hi all,

This idea has been shared with our internal teams and we are going to be researching it a bit more as we explore next possible steps—check out @kkim's comment in this thread for more info 💪

Feel free to continue the conversation here and stay tuned for updates.

Thanks!

[Jon]

Hi all,

This idea has been shared with our internal teams and we are going to be researching it a bit more as we explore next possible steps—check out @kkim's comment in this thread for more info 💪

Feel free to continue the conversation here and stay tuned for updates.

Thanks!

[Jon]

Hi all,

This idea has been shared with our internal teams and we are going to be researching it a bit more as we explore next possible steps—check out @kkim's comment in this thread for more info 💪

Feel free to continue the conversation here and stay tuned for updates.

Thanks!

[b00m3rh4ndz]

+1

[Arexodius]

Awesome!

[hpvd]

Microsoft Edge now offers/test support for upscaling videos with
- Nvidia and
- new: AMD
GPUs

see https://videocardz.com/newz/microsoft-introduces-edge-video-super-resolution-for-nvidia-and-amd-gpus

=> with this, So supporting all (modern) GPU types (discrete and integrated) from all vendors (Nvidia, AMD, Intel, see earlier post) in the near future looks reasonable.

[hpvd]

now AMD is onboard to:

https://videocardz.com/newz/microsoft-introduces-edge-video-super-resolution-for-nvidia-and-amd-gpus

[DarkJanissary]

We need Nvidia VSR support asap! I don't want to use data-stealer tracker-fest chrome bloatware.

[Saarsk]

FFmpeg already has support for this so this could be leveraged via that library, I'm sure.

FFmpeg has different types of upscaling, including machine-learning-based. The resulting upscaled image is quite impressive, actually (depending on configuration). This is on Linux, but I would guess that it has Windows-support as well? Not sure if it's as simple to implement on Windows.

GPU vendors probably have an driver API that can be leveraged too. Hardware-acceleration is quite important for laptop-users, for instance, to avoid fans spinning up, device heating up and using excess power just to watch YouTube or Netflix or anything self-hosted for that matter through the browser.

Anyway, I don't think this idea is far-fetched at all and as far as I know there are already different libraries that can easily be leveraged by FireFox (as it already uses FFmpeg).

[EvenBoe]

Definitely exited for a feature like this (if it is possible). I hold nothing against high performance, opensource, quality improvement! 😍

[SnowDrifter]

Would also love to see integration with Nvidia's upscaling.

Seems unfair to be left out of the chromium market monopoly

[BVEsun]

It would be better if it could support AMD as EDGE does.

[Invay]

Go add Nvidia VSR feature if possible. Thumbs up for that!

[Rari]

This would be massive for me if it was added.

[cutiepuke]

pls

[PBL]

Tested NVIDIA VSR in chrome, it's great and I've been waiting for such a feature for quite a long time.

Now we need it in firefox, NVidia or another one, and the sooner the better. Actually, I've seen quite a few others who have tested it wanting to leave firefox for edge or chrome for this feature and I can understand them...

[BethofDeath]

My vote re the nVidia method: ABSOLUTELY NOT

My vote re the AMD FSR method: Sure, why not?

We should not be supporting technologies which cause so much wanton ecological disruption as "artificial intelligence". https://www.technologyreview.com/2019/06/06/239031/training-a-single-ai-model-can-emit-as-much-carbo...

Yeah, the model is "already made", so the cost is "already done". But saying that "we should go ahead and use it, to amortize the sunk cost" is short-sighted for several reasons.

• nVidia's AI-based method requires a newer, much-more-power-devouring and expensive GPU, due to depending on their proprietary Tensor core.
• This quickly renders anything older than the RTX 30 and 40 series e-waste. They've claimed they can make it run on RTX 20 series also, but it would require some refactoring -- which, afaik, they haven't done yet, and which is still an expensive card to many.
• Needing newer hardware from a specific vendor for the feature feels pretty opposed to Mozilla's  philosophy towards internet accessibility.
• If it's functionality detected at runtime, it's yet another whitelist to maintain, or large blacklist to carry along and compare against.

AMD's FidelityFX Super Resolution is a much more reasonable and crossplatform approach, based on Lanczos upscaling. There are already OpenGL filters for this, such as come with mpv by default.

Even if developers didn't decide to specifically implement FSR -- which officially requires a GTX 10 or RX 400 series card afaik, but apparently can run on other and older hardware -- it might be easier to "simply" hook in such a filter on video for spatial and temporal upscaling for videos below a certain resolution and framerate, and perhaps even create a plugin which bumps up still image filtering quality, to use Lanczos or Spline36. This would provide the widest compatibility running back much further than competitors' officially-stated support -- without needing Vulkan on Linux, either. Not sure that's needed, I only found some Reddit post saying FSR only worked on Cyberpunk 2077 under Vulkan, and that's several levels of compatibility software involved...

I'd strongly advise performing tests on hardware performance/load and power consumption, however, and a check to not upscale videos which are already at or near the limits of the display hardware. No temporal upscaling for 60FPS videos on a monitor set to 48 or 60FPS, no spatial for 1080p on a 1080p display, viewport size checks, all that.