Creating an H.264 stream from Unity Content

Argus Simulator
Currently I’m working on a project at work that requires the information (lets call it unity content) displayed on twelve screens in another room/building (a long way away from the computer generating it), it’s a little bit of an unrealistic-solution/challenge to connect a dvi/hdmi/<or you name it> cable through buildings… as quality degrades the longer the cable gets… and not to mention all other obstacles along the way.

Enter streaming over ip… now this is not the easiest thing to do there are many ways to go about it.

The first option that came to mind was lets get a hardware adapter that can convert the DVI output to an H.264 IP stream.. that is a pretty expensive solution if you have twelve outputs that need to go to twelve screens, average cost of a decent hardware encoder runs somewhere around and above 4000 euros that times 12 and you have yourself an expensive streaming setup..

So Ideally thinking about costs the best solution would be do it in software, the solution first thought of was to do screen capture and encoding on the fly, all possible with vlc and many other software screen capture programs. After a few small tests this proved to be really CPU hungry and not really the best solution, because we need as much CPU & GPU time as possible for generating the Unity Content.

So now almost at the end of all options a colleague of mine mentioned AMD’s Radeon Sky Series which is basically cloud gaming tech not yet available for the masses. This got me thinking and looking more closer at cloud gaming which is basically a server on the web streaming games to specialized clients. Looking further into this I found Nvidia and the new range of GPU’s with Kepler Architecture *hallelujah* each Kepler based card is fitted with a beautiful h.264 encoding chip. Which turns out to do encoding without even touching the GPU or CPU time, and mighty fast too… 76 frames at 600ms which boils down to about 8ms per frame  which is what i’m looking for. with Low Latency settings I even got a 1.08ms encoding time… *jaw drop*

So I dove head first into the reasonably clear Nvidia API and expanded on my previous experience in creating a decent C++ native Unity Plugin. After bumping into alot of “unresolved external symbol” errors (which by the way usually mean your missing a lib file) and a few unicode/ascii hurdles (Yes i didn’t think this would be an issue, but it turns out C++ is picky about string encoding) I eventually got some output from the chip even though it was just a green screen.. it was still output and the encoder had been initialized to create it. Which meant progress!

Realizing that the green screen wasn’t my actual unity content I figured I’d better dig a little deeper and found that the green screen was actually blank YUV pixels, eventually I managed to change that to red using different YUV values and filling the texture with them. Still not so useful because Unity doesn’t output YUV pixels now does it. This led me to search the internet for a way to convert RGB to YUV (actually BGRA to YUV444) pixel for pixel conversion was an option with readily available formulas online.. only not the fastest way.
Browsing through the built-in shaders I found a reference to a YUV conversion, using that as a reference I created a new shader for myself to do the YUV conversion with some modifications to values and the order they are stored in.

Now a little under half a year later i’m using my plugin to encode 12 H.264 streams to send to VLC which passes it on to a VMS.
Video Will soon follow 😉

Project: MOBZ

For the past year (2011) I’ve been working at Movares on a simulator for the training of bridge & lock operators. Operators of what??? the people that control locks/sluices for boats and plus some bridges.
More info about MOBZ can be found here and here
I now know a lot about them. I built the interface and logic behind the simulator all of it was done in the Unity Engine. Check out this video to see what is all about (sorry the voice is in dutch can’t help you there).

Video Animations

As I’m a fanatic video and audio person I like to experiment with various applications and tools. I like to endeavour into unknown territory, learn a bit and move on to new horizons ever expanding and not always getting extremely deep into one thing. Below are a few animations I once made. I may add to some of these with what I now know sometime in the near future. First I want to capture some interesting video fragments to add stuff to.


Continue reading “Video Animations”

Photography Assignment

Once upon a time a few years back, I did a 10 week photography minor at Uni, These pictures were part of an assignment, they are a few of the best images I took during that course. Including the photos I took with a Pinhole camera.. I do miss doing that.. it was fun and gave unexpected results.. and it was real photography at the core 🙂

I mostly enjoyed spending time actually going out and standing in one place for 10 to 15 minutes to get enough exposure, plus it was a major kick to see the actual result after developing the photo, also sometimes a major dissappointment… I remember standing outside in the cold for around 20 minutes with my paint can duct-taped to a streetlight.. only to find out when developing that i had nothing but extremely dark patches…. so be warned 😉 Continue reading “Photography Assignment”

Unity GLSL Shaders

Hello Again,

In an earlier post of mine about WebGL I describe a few methods to enable WebGL on a Mac (Safari) the reason I did this was to view a few interesting shaders.. (is now Now comes the cool part..
I actually got those same GLSL shaders I wanted to view running in Unity (yes for me that is a big thing).. Check out a few of these screenshots I took… It wont be long now and I may even integrate them into Image Effects on Cameras.. Continue reading “Unity GLSL Shaders”

Unity Cornell Box

For anyone interested in the Cornell Box and rendering it this is my attempt at it in Unity…
First of all I acquired the geometry for the cornell box from the site in 3DS Max exported it as a FBX file (available below) then opened up Unity3D. Using a the jpeg available from the site. Continue reading “Unity Cornell Box”