Direct integration of NVIDIA NVENC encoder

[Xaymar]

Discussion Thread

#856

Provide a summary of the Improvement

The current (via FFmpeg) integration is not ideal, we should aim for a direct integration of NVIDIA NVENC.

Completion Requirements

• A reasonable amount of documentation should be present in the code.
• Support for Windows and Linux with the available Graphics APIs in libOBS.
• Zero-Copy and Multi-GPU encoding should be supported with the best matching API (if possible on the API).
• Both H.264/AVC and H.265/HEVC should be available for encoding.
• The User Interface should expose all useful UI options in a similar way to the (via FFmpeg) encoders. This includes the Custom Settings field, which allows users to specify settings that override everything shown in the UI.
• All common encoding formats between libOBS and NVENC should be supported (NV12, I444, RGB) in the bitness available at implementation time.
• Performance should be similar to the already existing (via FFmpeg) NVENC encoder, if not better. Memory usage is secondary and can be worse if necessary.