Creating an H.264 stream from Unity Content

Argus Simulator
Currently I’m working on a project at work that requires the information (lets call it unity content) displayed on twelve screens in another room/building (a long way away from the computer generating it), it’s a little bit of an unrealistic-solution/challenge to connect a dvi/hdmi/<or you name it> cable through buildings… as quality degrades the longer the cable gets… and not to mention all other obstacles along the way.

Enter streaming over ip… now this is not the easiest thing to do there are many ways to go about it.

The first option that came to mind was lets get a hardware adapter that can convert the DVI output to an H.264 IP stream.. that is a pretty expensive solution if you have twelve outputs that need to go to twelve screens, average cost of a decent hardware encoder runs somewhere around and above 4000 euros that times 12 and you have yourself an expensive streaming setup..

So Ideally thinking about costs the best solution would be do it in software, the solution first thought of was to do screen capture and encoding on the fly, all possible with vlc and many other software screen capture programs. After a few small tests this proved to be really CPU hungry and not really the best solution, because we need as much CPU & GPU time as possible for generating the Unity Content.

So now almost at the end of all options a colleague of mine mentioned AMD’s Radeon Sky Series which is basically cloud gaming tech not yet available for the masses. This got me thinking and looking more closer at cloud gaming which is basically a server on the web streaming games to specialized clients. Looking further into this I found Nvidia and the new range of GPU’s with Kepler Architecture *hallelujah* each Kepler based card is fitted with a beautiful h.264 encoding chip. Which turns out to do encoding without even touching the GPU or CPU time, and mighty fast too… 76 frames at 600ms which boils down to about 8ms per frame  which is what i’m looking for. with Low Latency settings I even got a 1.08ms encoding time… *jaw drop*

So I dove head first into the reasonably clear Nvidia API and expanded on my previous experience in creating a decent C++ native Unity Plugin. After bumping into alot of “unresolved external symbol” errors (which by the way usually mean your missing a lib file) and a few unicode/ascii hurdles (Yes i didn’t think this would be an issue, but it turns out C++ is picky about string encoding) I eventually got some output from the chip even though it was just a green screen.. it was still output and the encoder had been initialized to create it. Which meant progress!

Realizing that the green screen wasn’t my actual unity content I figured I’d better dig a little deeper and found that the green screen was actually blank YUV pixels, eventually I managed to change that to red using different YUV values and filling the texture with them. Still not so useful because Unity doesn’t output YUV pixels now does it. This led me to search the internet for a way to convert RGB to YUV (actually BGRA to YUV444) pixel for pixel conversion was an option with readily available formulas online.. only not the fastest way.
Browsing through the built-in shaders I found a reference to a YUV conversion, using that as a reference I created a new shader for myself to do the YUV conversion with some modifications to values and the order they are stored in.

Now a little under half a year later i’m using my plugin to encode 12 H.264 streams to send to VLC which passes it on to a VMS.
Video Will soon follow 😉

27 thoughts on “Creating an H.264 stream from Unity Content”

  1. Hello iKriz, I am faced with a need similar to yours in that I’m trying to set up a Unity plugin to encode a camera’s Texture2D to a H.264 video with Kepler’s NVENC.

    I have started using NVIDIA VIDEO CODEC SDK and have compiled the samples, however I still struggle getting things done from Unity. Would you be so kind as to share your source code, which would help me get on the right tracks? 🙂

    Thanks,
    Nicolas

    1. Hi Nicolas, Sorry from my late reply. I’m toying with the idea of releasing the code it belongs to my work officially. The whole reason I haven’t released it yet is cause it needs refining really, plus i want to know how the company stands towards releasing it. I’d be happy to answer any questions. I started working on it in DirectX 9 it’s now using DX11. Plus it could use more optimizing.

      1. Hello, no worries about the late reply, it forced me to dive deeper into the API 🙂 even though I kind of switched to other aspects of the project as of lately, I’ll have to come back to it at some point.

        I am having trouble passing a camera render texture from Unity to the plugin. My initial idea was to use texture.GetPixels32() and pass that as a pointer to see what I could do with it in C++, but it didn’t work. Does it ring any bell?

        Since you’re talking about DirectX, are you using the low-level native plugin interface ( http://docs.unity3d.com/Manual/NativePluginInterface.html )? If yes, how did you properly setup the encoder?

        I’m quite inexperienced with encoding stuff, so even if it’s not refined/optimized yet, and if you and your company are OK with that, I’d be more than happy to have a look at your code. Perhaps if you don’t wish to release it publicly until it’s fully refined we could do that by email. Just so you know, my project isn’t commercial, it’s for research purposes 😉

        1. I’m starting to not like my site template now… Indeed I developed a native dll that uses Texture.GetNativeTexturePtr of a RenderTexture (but a Texture2D is probably faster requires less conversion) load that in with directx then you need to do a bit of data conversion and can pass that to the encoder. I had a little contact with nvidia asking what settings are best to use and they came back with a load of optimizations which brought the encoding time way down.

          Getting the pixels through GetPixels32() could probably work maybe slower though, I kept all data transfers within the native plugin. Most complex part of it for me was starting from scratch and with only a basic knowledge of c++ 🙂

  2. Holy hell, that last comment went out even worse than expected.
    Copy pasting, you can delete the previous one I guess:

    ROFL. That template indeed is not suited for nested replies. I’ll play a bit with the native pointer when I come back to it.

    Thanks for your answer and if you ever have news from your company and can send me some code, in particular these NVIDIA optimizations, I’ll have a look 😉

  3. Hello, so I’m back on my code.

    I have prepared a RGBA to YCbCr shader to use with Graphics.Blit() before sending the RenderTexture to the encoder using Texture.GetNativeTexturePtr.

    However I can’t seem to load the returned ID3D11Resource and encode it. How do I properly pass this resource to be processed by the encoder, i.e. what am I supposed to do right before nvEncEncodePicture()? How do I fill the input surface? I’ve been stuck on this two days and can’t seem to get my around it…

    If you have some time to help it’d be wonderful 🙂 You can contact me by mail if you prefer or if you need to attach some code.

  4. Hello iKriz,

    Excellent work! And, thanks for writing about it. I am about to do the same thing. Plus, I need to write custom apps to downstream the video to iOS, Android, Win, and MacOS. I would be quite happy to help you solve your issues. If you don’t mind I can send you an email.

    Thank you!

    Geo

  5. Hi iKriz,
    I am able to get H.264 stream working from a server to client using Texture2D but the image is flipped. Any idea what is going wrong?

    Also when I use RenderTexture as render target nothing is captured. Can you please provide some input on how you got this working.
    Thanks,
    Rishi

  6. Dear Ikriz,

    I’m also looking for Creating a H.264 stream from Unity Content. Do you have something available?
    Greetings
    Douwe

  7. Hello iKriz,

    very interesting post. I would be grateful if you could answer some of my questions. I’m also looking into streaming rendered content over a local network.

    Actually I’m amazed that you got your plugin to encode twelve! streams. What kind of GPU do you use for the rendering and encoding? I was under the impression, that those numbers can only be achieved with high-end cards like Quadro, GRID K… and the like?

    Also what was the overall latency you achieved with this setup including network latency? Did you optimize for real-time interaction or was that not relevant for your intended use?

    Thank you very much.

    1. Hi Julien,

      Sorry for the late reply, Indeed optimized for realtime interaction, got it down to a fast enough reaction (I believe is was around a second) eventually after removing a few buffers in the chain. Using Quadro K5000 cards (currently outdated though) 12 streams was just a matter of scaling.

      Kinda want to make an open source plugin which would help the community aswell, just need to convince work.

  8. Hi Kriz,

    If you can’t give it away for free can you then let me know what the cost will be? I think it would really help us.

    Greetings,
    Douwe

      1. Hi Kristen, sorry I think I typed me email wrongly in previous comments. Can you reply me too? I’m interested in discussing a commercial deal.

  9. Hi Kriz,

    Excellent work! I’m also looking for something similar. If you can’t give it away for free, would you be so kind as to share your source code by email. I think it would be great and will really help us. I’m a postgraduate student, and my project isn’t commercial, it’s for research purposes.

    Greetings,
    Sylvester

      1. I’m sorry, may be I didn’t express it clearly. I find that you send the email to Nicolas B and Douwe, so I ask you for help. I just want to know how you convert the data from ID3D11Resource to the encoder.And whether you’re using NVDIA VIDEO CODEC SDK to creating an H.264 stream?
        This question plagued me for a long time, so if you can give me some help, I will be so thankful!

Leave a Reply