Creating an H.264 stream from Unity Content

Argus Simulator
Currently I’m working on a project at work that requires the information (lets call it unity content) displayed on twelve screens in another room/building (a long way away from the computer generating it), it’s a little bit of an unrealistic-solution/challenge to connect a dvi/hdmi/<or you name it> cable through buildings… as quality degrades the longer the cable gets… and not to mention all other obstacles along the way.

Enter streaming over ip… now this is not the easiest thing to do there are many ways to go about it.

The first option that came to mind was lets get a hardware adapter that can convert the DVI output to an H.264 IP stream.. that is a pretty expensive solution if you have twelve outputs that need to go to twelve screens, average cost of a decent hardware encoder runs somewhere around and above 4000 euros that times 12 and you have yourself an expensive streaming setup..

So Ideally thinking about costs the best solution would be do it in software, the solution first thought of was to do screen capture and encoding on the fly, all possible with vlc and many other software screen capture programs. After a few small tests this proved to be really CPU hungry and not really the best solution, because we need as much CPU & GPU time as possible for generating the Unity Content.

So now almost at the end of all options a colleague of mine mentioned AMD’s Radeon Sky Series which is basically cloud gaming tech not yet available for the masses. This got me thinking and looking more closer at cloud gaming which is basically a server on the web streaming games to specialized clients. Looking further into this I found Nvidia and the new range of GPU’s with Kepler Architecture *hallelujah* each Kepler based card is fitted with a beautiful h.264 encoding chip. Which turns out to do encoding without even touching the GPU or CPU time, and mighty fast too… 76 frames at 600ms which boils down to about 8ms per frame  which is what i’m looking for. with Low Latency settings I even got a 1.08ms encoding time… *jaw drop*

So I dove head first into the reasonably clear Nvidia API and expanded on my previous experience in creating a decent C++ native Unity Plugin. After bumping into alot of “unresolved external symbol” errors (which by the way usually mean your missing a lib file) and a few unicode/ascii hurdles (Yes i didn’t think this would be an issue, but it turns out C++ is picky about string encoding) I eventually got some output from the chip even though it was just a green screen.. it was still output and the encoder had been initialized to create it. Which meant progress!

Realizing that the green screen wasn’t my actual unity content I figured I’d better dig a little deeper and found that the green screen was actually blank YUV pixels, eventually I managed to change that to red using different YUV values and filling the texture with them. Still not so useful because Unity doesn’t output YUV pixels now does it. This led me to search the internet for a way to convert RGB to YUV (actually BGRA to YUV444) pixel for pixel conversion was an option with readily available formulas online.. only not the fastest way.
Browsing through the built-in shaders I found a reference to a YUV conversion, using that as a reference I created a new shader for myself to do the YUV conversion with some modifications to values and the order they are stored in.

Now a little under half a year later i’m using my plugin to encode 12 H.264 streams to send to VLC which passes it on to a VMS.
Video Will soon follow 😉

Unity Media WordPress Plugin

For all of those people out there with a wordpress site and a will to show some unity content. Here’s your chance to shine…
You can use the plugin below to do exactly that, glad to be of assistance.

Unity Media Plugin

Unity Media Plugin allows you to upload various Unity Engine related files to your WordPress site. It also allows you to add a Unity Webplayer to your post.

Major features in Unity Media 1.0 are:

  • Uploading of FBX files, .unitypackages and .unity3d files
  • Inserting multiple webplayers into your posts

Download:

You can download my plugin here: u3dmedia.zip (919 downloads)

Quickstart:

1. Upload plugin zip to site
2. Activate plugin
3. You can now upload a .unity3d file
4. And place the following tag in a post to add a unity webplayer to it:

[UnityMedia src="<url of unity3d file>" width=550 height=344]

Let me know if you like this plugin or if you think it’s missing some important feature, your welcome to comment or donate so I can update it.

Hello 2013… Happy New Year Everyone!

2012 has been a great year with a cherry on top… just before new years on the 27th my wife gave birth to our brand new daughter who is doing very well. This last year has been good, I got a permanent contract at work and we moved into our own house back in feb, all in all it’s been a rather eventful year. Of course the last few weeks have all been baby filled, now with things settling down and gradually getting in to the rhythm of broken sleep and changing diapers there is time in between to do grown up stuff.

What grown up stuff is there to be done? you may wonder?
[Tech Talk]
Well I noticed a few things on my site that needed fixing so i’ve tended to that, you’ll now notice that all unity webplayers on the site should be working much better. Plus i’ve updated the theme template to the newest version. Not to forget updating wordpress which was well over due.

In the near future I’m playing with the idea to switch hosting as this is getting to be a little over the top for something not generating much in return. If your reading this now your welcome to donate if you like what I do here, or if you have any tips for me please comment.

I may offer some nice stuff to the Unity Community via the Asset Store once I’ve figured out what that could/should be. If you have any requests or ideas for what an experienced Unity Developer should make and you’d be willing to pay a nice sum for it let me know.
[/Tech Talk]

Other than all of that technical stuff, I’m looking forward to the remainder of 2013 and what it has to bring, as well as watching my daughter grow and build her character. I do hope everyone enjoys 2013 in good health and free of negative stuff, so.. for now I wish you all a happy new year!

Pro Tip: Don’t let the bed bugs bite and steer clear of smelly diapers.

_MG_5037

Unity Post Process Mayhem

Are you sick and tired of forgetting to copy that data file to the build directory after each and every standalone build in unity? Sure you could incorporate it into the project but you don’t want that, you just wish that PostProcessBuildPlayer script would work on windows. As of Unity 3.5.2 life just got better…

A long awaited function in Unity was a working post process build player script for standalone players on windows as it was already working on mac’s. As seen in the latest Release Notes and stated in the BuildPipeline a few things have changed.. Apparently the PostProscessBuildplayer script is not supported on windows, which is a bummer for windows users on the other hand it’s easily solved with a new and improved UnityEditor.Callbacks which is both usable on Windows and Mac, So here goes…

P.S. No this is not a custom build window script, it actually triggers from the existing Unity BuildWindow.

NB: -10 is a higher priority than 100, the default priority is 1
NB2: This is an Editor Script not something for a gameobject (you should know this)

You can use the method to do all kinds of things on your computer after a build is complete just a couple ideas:

  • After a large build send an email with a link to the file saying the build is complete.
  • Play a sound notifiying you build is complete
  • Run a test on your build
  • And lots more…

Anyway now you have the tools go make 🙂

Unity Editor Macros

A while back the guys at unity released a version of their editor with macros enabled (by accident) in the current version is has been removed/disabled. (this because it’s something they use for testing and not meant for production environments)
But with recent endeavours I found it’s possible to enable it in the editor making debugging stuff within the editor more entertaining, plus it adds a new window to the editor making your editor more of an editor.
Please note this is not documented by unity and probably not supported either.

Simple script:

How to use:
Place this script in an Editor directory in your Assets and then:
Go to Window->Macros and tadaa.. you have a new editor window that can record all your editor actions and execute new ones.
May this help you on your endeavors of making cool new stuff for unity

Project: MOBZ

For the past year (2011) I’ve been working at Movares on a simulator for the training of bridge & lock operators. Operators of what??? the people that control locks/sluices for boats and plus some bridges.
More info about MOBZ can be found here and here
I now know a lot about them. I built the interface and logic behind the simulator all of it was done in the Unity Engine. Check out this video to see what is all about (sorry the voice is in dutch can’t help you there).

Unity Video Remake: Webcam Textures

A while back I posted something about video capture in Unity with a sad plea for beta access to version 3.5 which is now finally in public beta, so I decided to do a new post.

Setting up a Webcam Texture in Unity

Now as you may have seen in the Script Reference (the only place to actually find something about a WebcamTexture) there is no Object like a RenderTexture for a WebCamTexture which makes it a tad more harder to work with (read: not as easy to drag and drop in to materials).

Quickest way to get this working is to drop the following script on an object.

Checkout this script in action Webcam Example

Keep in mind that this in not the best way for multiple objects unless they all have the same material with the script on one object.
You can only have one WebCamTexture per webcam!

By the way.. Once 3.5 goes fully public I’ll put up a working example to give you a starting point, and so you can see yourself (if you have a camera connected of course) so keep an eye on my site 😉

If you’d like more help or a better example please donate any amount you wish and i’ll get to it..

Once you’ve got the camera running it’s possible to do all kinds of things.. I Look forward to the future of Webcams in Unity all over the internets. 😉

Issues: Cannot use web cam, since the user has not authorized this!

In your attempt at getting the webcam running you may have come across the following error: (I did at least)

Cannot use web cam, since the user has not authorized this!
UnityEngine.WebCamTexture:Play()

You can squash this little one by changing your build platform to Standalone Mac/PC. Go to Build Settings -> Select “PC and Mac Standalone” then click that little button at the bottom left “Switch Platform”, this, I think is something new.

The whole reason for this is because you probably have the Web player platform setup by default. This requires the user to first give permission to use their webcam because of privacy reasons. If you want to do this then you need to request the users permission by using the following piece of code:

IEnumerator Start()
{
yield return Application.RequestUserAuthorization(UserAuthorization.WebCam);
if(Application.HasUserAuthorization(UserAuthorization.WebCam)) Debug.Log("Authorized");
}

Same goes for UserAuthorization.Microphone. When you use a standalone platform the authorization is no longer needed.

Issues: The type or namespace name `WebCamTexture’ could not be found.

Thanks to solehome for commenting on this one. The public beta 3.5 seems to have an error when using WebCamTextures in Flash builds.
If you choose to build on a flash platform the following errors occurs:

error CS0246: The type or namespace name `WebCamTexture’ could not be found.

Are you missing a using directive or an assembly reference?

Error building Player because scripts had compiler errors
Solution to this problem… euhm dont build flash?

[ad]

Unity Video Capture

There once was a Unity Engine roadmap without any means of capturing video or audio, conclusion: If you wanna do it right do it yourself or hope unity implements it themselves.. w00t v3.5 🙂 well anyway before 3.5 my little plugin named Video Grabber was born…

Built on the basis of OpenFrameworks (being one of the frameworks to handle video for windows & mac at the same time) and also allowing for alot more fun.
I successfully built this library as a plugin into Unity. In doing this I can now capture pixels from the camera and transfer them to a texture in a scene. I even added in some face recognition to the whole shebang…
It works quite well with a slight delay of a few milliseconds, but who really cares about that. BTW oF is doing some really cool stuff 🙂

So what happens when you’ve almost finished a functioning Video Grabber (except for a couple of problems when stopping the plugin) you find out that Unity is going to implement it themselves.. of course.. Unity wouldn’t be Unity if they didn’t.. you need video we’ll put it right in there, how about some audio as well then.. whut? you want AI pathfinding… sure no problem. While we’re at it we’ll mix in a little UI update and some extra goodies… What can we expect in version 4.0? A “Generate AAA Game Concept” buttonâ„¢ all ready and waiting?

One thing’s for sure.. I’ve temporarily stopped working on my Video plugin with high expectations of Unity 3.5 coming.. 🙂
[insert plea to beta group owner for a membership here with a cherry on top]

If your interested in looking at what I’ve currently got for video grabbing add a comment below make sure to add your email address in the proper field (which will stay private) i’ll add you to the private repo and we maybe could start an open source project for the community. 🙂

[ad]

Gameplay: Starcraft II Wings of Liberty

Just finished the Wings Of Liberty campaign this weekend. Good story and I like the gameplay around it. It be cool to know a bit more behind the scenes and how on earth they created such a game, in terms of game logic and mechanics behind it. It must be one huge amount of variables and such from a programmers point of view. I do like the “Thanks To” part in the credits.. “Your Mother” 😛 some smart aleck put that in there. It’s really the only part I took the time to read, because it seems to genuinely come from the developers.
Anyway, I enjoyed playing it thats for sure.. So now I’ve also played a bit against the A.I. on “Medium”, I’m missing a Normal level.. Medium seems to be just a little bit on the easy side. Next game I’ll go for Hard, hoping I can still win.
Your thinking I could also just play online against someone.. well yes thats true I could get my ass whooped by some guy that only plays online all day and end up losing and getting all depressed about it… I like my #winning 🙂 Currently my strategy against A.I. is to teister it a bit and then come a long with a huge amount of a few types and just waste the **** out of them, most of the time it works too. It sure helps blocking the way in the beginning, otherwise I do lose.

By the way the visuals in some cutscenes are just amazing I’m stunned by the way they did the rain flowing over the metal. That is some really nice work! (even though it is the Havok engine and i’m all into Unity at the moment) I’m guessing thats a custom made shader of somekind, well it cant be default Havok stuff right?