Unofficial OVRLipSync Plugin for UE4

After messing with PocketSphinx’s phoneme recognition and not getting the results I wanted, I looked back into porting Oculus’ LipSync plugin for Unity over to UE4, since the Unity plugin is just a wrapper for a DLL and some examples of how to use it.
I’m happy to report that I’ve got a basic version of the OVRLipSync plugin working in UE4, and it’s ready for people to use. I’ve gone ahead and made an example project to show how to use the UE4 version of the plugin (quite straightforward, see example images below). The project has an example mesh to see it in action, and should work out of the box:

Example Project + Plugin Repo:

Plugin Only Repo:

For those that just want a quick rundown on how to use it without downloading the example project, here are some screenshots of the VisemeGenerationActor derived blueprint class.

VisemeGeneratActor event graph setup:

SetMorphTargets function:

MorphIdxToNames array:

As long as the mesh has the appropriate morphs as listed above, it will work decently well. I’m sure there are things I’m doing wrong, and bug reports are welcome!

Floating Hands: Good Riddance

I never liked the idea of floating hands, and now that I’m no longer trying to get a dev kit from HTC/Valve (they expressed interest in mini-games with low replay value and minimal content, the opposite of what Dungeon Survival is aiming for), I’m doing away with them entirely in favor of the true-first-person I always intended to provide, and was going to end up being an option if floating-hands had stayed.

With an inverse kinematic solution that’s more robust than the generic CCD/tw0-bone IK provided out of the box by most engines, and which also features constraints, the “elbow problem” which prompted Oculus and Valve to so vehemently recommend against true-first-person avatar bodies simply doesn’t exist, so the argument for floating hands crumbles.

For fans of floating hands rather than a tracked, full body avatar (why!?), they may come back as an option post release.


Belated Screenshot Saturday – Dungeon Survival


About Dungeon Survival

Features Overview

  • Made from the ground up for VR, with unique features to support motion controls
  • Next-gen VR features today: Voice-input commands, NPC facial morphs and more.
  • Co-op multiplayer (at least 2-4 players, maybe more unofficially supported)
  • Streaming, no-load-screens transitions between levels of the dungeon
  • True First Person (you can see the character’s body, including animations)
  • Deep Item System – Fire burns things, water douses them, etc. – generalized gameplay that layers up to produce complex interactions
  • Real-time Physics Interactions and Combat – Items are physics based as in games like Skyrim, combat is based on physical collisions – if you see a hit – you hit!
  • Environment Interaction Over Hack ‘n’ Slash – Use your wits to overcome the dungeon’s deadly foes and traps! Ingenuity is rewarded over blunt force.
  • A Customizable Experience – Want hardcore roguelike style gameplay with permadeath? Or would you rather have a more Skyrim-style experience? You choose!
  • Modding Support – Via both configuration file editing (recombine base gameplay of items/monsters with new looks and combinations of behaviors) and advanced modding through the UE4 engine. Modding means for a replayable experience you can keep coming back to for years to come.

About The (VR) Dungeon Survival Project

The (VR) Dungeon Survival Project is a game built out of my desire to make something that captures the magic of deep item and environment interaction gameplay of roguelikes, on top of a foundation of survival and real time gameplay.

In addition, the game is my experiment into VR and supporting novel input devices for more interactive, fine-grained gameplay. What this means is that ultimately Dungeon Survival will support setups that allow the player to play with natural input, such as grasping and manipulating objects with the HTC Vive controllers.

Mod support is also very important to me, as I fully understand that mods are the key to longevity of games of this sort. Most things will be able to be tweaked or added to without any tools other than a text editor, but advanced functionality will require downloading a mod pack with source files for the game, and getting access to Unreal Engine 4.

Grass Bending Test

Note: The following video is pre-alpha test footage, and in no way represents the quality of the final game.

It’s hard to see due to the video framerate and compression, but there are volumes for affecting the grass parented to the player’s feet.

Credit for the effect goes to Daniel Wenograd.

To add this to your own Unreal Engine 4 project, check out Daniel Wenograd’s post on the UE4 forums.

Progress Update Since Last Time

Dungeon Survival has progressed a lot since the last update. No screenshots currently, but I would like to describe some of the features I’ve completed.

  • Procedural dungeons!
  • Zombies with animation (attacks, laying, sitting, hits etc) and AI, and swarm avoidance using detour and recast. Zombies wander and will detect and chase players, and attack when close enough.
  • Upgraded to UE 4.7.6.
  • Added several traps, as well as ragdoll assets for the player and zombies, and code to ragdoll on death.
  • Early melee animations

Motion controls for the arms/hands are working great with the Rift, and I’m eager to put in Vive/SteamVR support. The game is naturally suited to moving around a room, and I have some ideas for handling long distance locomotion.

Being able to bend over to reach items on the ground is already amazing, but I’m quite limited by the Razer Hydras I currently support.

More work is going into levels and traps, as well as some additional enemies to add to the rats and zombies. Eventually a character creator is going in.

I’ve also found and plan to implement physics rope bridges, and a system for detecting player breathing which will translate into character breath in game.

Despite infrequent posts here, development continues to progress at a good clip. I’m currently waiting on certain art assets to become available before the next level design phase takes place.

Porting up to Unreal Engine 4.3 – Setbacks Caused by Boneheadedness

Click image for source! (Credit goes to the artist! Not affiliated with the project - used for blog enhancement purposes)

Facepalm – Whiterun Guard


Progress has been swift but not without setbacks, as with the problems I encountered moving my project from UE4 4.2.1 to 4.3 preview.

I decided to port up to it because it offers some particular functionality I required, namely the On Actor Fracture event.

I use this to figure out when the destructible oil barrel has cracked open, to spawn the oil spill (which as described previously (under Other Stuff That’s Done) – can be lit by throwing a torch onto it, or other on-fire item). There are some other improvements I wanted access to, but that was the reason for switching now.

Unfortunately I made a bonehead move and converted a copy of the project, started working immediately on adding new stuff (torch throwing, barrel breaking spawn oil spill etc.) without actually checking to make sure that everything worked properly otherwise.

It did not.

The plan for last night had been to take a short video of how the gameplay with the oil barrels and throwing torches was shaping up, but instead…

I spent last night debugging the things that were wrong I had missed, mostly related to the item pickup/drop functionality over the network. My plan for today is to finish up going through the 4.2.1 source blueprints and make sure everything is intact and correct.

What seems to have happened is that the conversion didn’t actually get all the blueprint nodes and variables, and in some cases changed the options of other nodes (raycast was set up to show debug, despite debug being off in the 4.2.1 project).

Once everything is back to normal in the 4.3 project, I’ll be working on overhauling the inventory system.

Here’s the Trello task for that:

Inventory Overhaul Task on Trello

Inventory Overhaul Task on Trello

Once that’s all finished, it’s on to setting up the rat enemy and then the AI for it.

Rats! And You, In Dungeon Survival

Rat Swarm by Jay Odjick

Rat Swarm by Jay Odjick – (Not affiliated with the game – just for looks! All credit goes to the artist!)

Rats are usually boring enemies in fantasy and medieval games, used to introduce new players to combat, and hardly intended to inspire terror. The exception is usually some sort of giant rat, but that too has limited scare factor when the players are used to fighting giant-everything (spiders, wasps, bats, etc.) any of which is more creepy than a giant rat.

In Dungeon Survival, I want rats to be the first enemy players encounter, not because they will be pushovers, but because they will have some interesting mechanics that will heighten their fear factor and hopefully, do justice to how creepy a bunch of rats in a dungeon would be.

In this post I’ll be going over two aspects of rats in the game, though the first will be about the player.

The Kick Attack

Your character will feature a few basic unarmed attacks. I’ll be going over the kick attack now.

* Physics based – It will work by having a physics volume follow the animation of the kick on the character.
* When the volume connects, it will impart damage on the hit actor, as well as an impulse. Any object that responds to physics will fly out.
* Enemies that take enough of an impact on this stage will go into ragdoll mode.

Rat Behavior

One thing that almost anyone instinctively fears is disease, and in the case of rats, that means them crawling onto you.

* Rats will be able to crawl up the character.
* Using hand motion controls (or a contextual punch attack – more on that in another post) the player can slap the rats off, but it has to be done with enough (in game) force to actually move them.
* Similar to the kick attack, a strong enough hit will make the rats ragdoll.

In addition to that, rats and other enemies will take impact damage – so a good kick off a ledge can be a one hit kill if done properly.

(Those readers interested in VR will note that the rat climbing behavior outlined above will be particularly creepy in VR. This is by design.)

Might look something like this.

Swarm of Shadow Rats by Christopher Burdett (again - not affiliated with the project, this is just to illustrate what I mean!)

Swarm of Shadow Rats by Christopher Burdett (again – not affiliated with the project, this is just to illustrate what I mean!)

The Dungeon Of Death! (and gameplay)

In Dungeon Survival, we must provide a counterpoint at all times to the mundane aspects of survival, maintaining one’s health in an environment of slimy subterranean walls, and avoiding the denizens of such places (or dispatching them with one’s ingenuity).

I am speaking of course of the all encompassing dread reaper: Death!

In other words, the Dungeon Survival project now features death and respawn.

In addition to this, since the last update (which was somewhat out of date) the game has:

Inventory System

This is sort of the basis for much of the game, as most of the objects will be “smart” – with some gameplay associated with how they work or interact with the environment/other objects in an intuitive way. I’ll follow up with a good example of this in a minute.

Items when on the ground are physics objects, much like in the Elder Scrolls series. When picked up, the object is attached to the player’s hand.

Currently, only one item can be carried at a time, not counting the “utility” slot (the right hand – which is at the moment occupied by the torch and isn’t completely hooked into the inventory system). In future, the player will be able to craft pouches which attach to the player and will be able to carry certain amounts of items of their own, depending on size/type.

As with the rest of the game, all of these things are networked for multiplayer – including kicking around physics objects (this may change depending on if it presents problems later) and having it sync to everyone.

When picked up, the item is not destroyed – it is taken into the character’s inventory, and attached directly to the hand (or will be “stored” in pouches if available). This is important because the item system in the game will also be persistent – a half-eaten apple will stay half-eaten even when dropped (and as that example implies, the player does not eat food all-at-once). Items will have persistent state.

The above behavior is for the default item type. All specialized item types derive from this type of item and include its behavior (though in special cases they may override it).

As of yet there is only one specialized type – the food item. Which brings us to…

Hunger and Thirst

First, some rationality for even including what some might consider a micro-management gameplay feature.

I’ll always remember my first few forays into the realm of Nethack, that character-based dungeon crawler with timeless charm and a rich set of gameplay.

In particular was when I first made it to level 3 (this was probably not Nethack itself but a derivative, so forgive inconsistencies, those Nethack aficionados in the audience). I had no food. I was dying of hunger, and a dwarf attacked me.

I dispatched the little fellow with little health left to spare, crawled around a corner and took the only option that remained to me. I ate the dwarf.

I ate his corpse and promptly died from dwarf-poisoning.

For those that don’t know – Nethack and its like feature “permadeath”, meaning you must start over if you die. Instead of being (justifiably, since the 3rd level of the dungeon was the farthest I’d ever progressed) irritated, I was cracking up with laughter.

I loved the fact that I was able to improvise in that way and eat the body of a dwarf I’d slain. And even more that it distinguished poison from normal food.

That’s why I wanted to make the Dungeon Survival project in the first place – to capture the magic of the “emergent gameplay” that roguelikes provide, but with a focus on natural interaction with the environment, and on survival gameplay (with, of course, VR heavily supported).

What’s done:

* Hunger ticks town every 5 seconds (numbers here are subject to change with testing).
* When hunger runs out, the starvation meter kicks in. It ticks down ever 5s as well, but at a different rate (you have more starvation meter than hunger meter, but it depletes in larger chunks).
* When your starvation meter runs out, you start to take damage to your health.
* As mentioned before, food items exist and can replenish your food meter.

Thirst and dehyration work in a similar way, but are not hooked up at the moment to aid in testing (and because I haven’t hooked up any drink items, though that doesn’t take much time).

To come:

* Cooking, and uncooked food causing sickness.
* Starvation and eating too quickly after having starved for a while will also cause sickness.
* Poisoned/contaminated food.
* Dehydration and drinking too much water when dehydrated causing sickness.

Other Stuff That Works Now

In addition to all that, I’ve added:

* A proper blend for the torch hold animation, so that you can right click to sheathe/unsheathe the torch. The character’s arm will smoothly blend up and down.

* An oil slick blueprint, which includes an oil decal, trigger volume and fire effects.
* Code so that if the player walks over an oil slick with his torch out, it will light on fire (and does damage).
* A destructible barrel – damage accumulates over time, so that if thrown/hit against stuff enough it will shatter.
* An oil barrel version that spawns an oil slick when it breaks.
* A torch physics item using the inventory system, and the ability to throw the torch via clicking without anything in held in the left hand inventory slot.

This is the sort of “smart object” gameplay I referred to before. Eventually things like the barrel spawning an oil slick, the slick being flammable, and items that have fire lighting the slick up will be generalized so that items will simply be able to specify what types of smart gameplay they use (by both deriving from special item types and settings in those types).

The basis for this system is already in place, and is how the food items work – but currently the torch and barrel do not use the system (that will change).

To come in the short term:

* Thrown torches will light oil slicks on fire.
* Throwing your torch will remove it from your hand,
* Being able to pick up the torch to re-equip it to the utility/right hand slot.
* A two-hand pickup system for picking up things like the barrel (luckily though I’m no animator, I’ve figured out how to build an animation for that and make it work myself).
* Being able to throw two-hand pickup items.
* Kicking/punching physics objects, like the barrel.

After that, I’ll be adding the first enemies to the game (rats!) and building out the first level of the demo dungeon.

Soon I’ll be taking some screenshots and video of the oil barrel gameplay.