To go along with the theme for UE4 game jam "All's fair in love and war" we came up with an idea for a family trivia board game "Hill Top Trivia".  We wanted to push the boundaries rather than play it safe for the jam so we decided to enhance our game with "AI" and voice driven mechanics.  This is obviously easier said than done.  Any search on the UE4 forums about voice recognition will yield you a ton of results, but not so much in the way of dedicated plugins or guides for implementing such technology in Unreal Engine.  We have products driven by voice, but these are mostly powered by Linux/Python.  With this in mind we decided to use a fork of the Unreal Engine Python plugin in order to accommodate the Speech To Text processing needed for our game. 

This strategy worked very well when running the game inside our UE editor, but packaging with Python is known to be touchy so in the end our packaged game was pretty much "semi functional" since by the end of the 5 days our game was almost completely voice driven, but not so much if you try to download the current version (but wait, there's hope!).

We have been communicating with PhonalAI which is bringing some great voice interaction tools to both Unity and Unreal Engine.  Their tools should provide the missing piece for our game to create a game experience that is fun, and social in nature.  The great thing about building our game with Unreal Engine is that we can easily plug in better voice interactions without rebuilding other core mechanics of the game.  Voice technology is advancing rapidly and hopefully we'll be able to gradually enhance that part of the game as better system becomes available in the near future.

Here is a quick breakdown of the other technology that powers Hill Top Trivia.

Text To Speech

We have taken advantage of some of the latest available TTS technology and applied it to speaking trivia questions out loud.  This means we were able to provide a large set of trivia questions without needing to do voice over work for every question. 

Trivia APIs

The text to speech system dynamically pulls trivia questions from an online trivia API,  Which means we did not have to build a huge question and answer database directly into the game.  With the current blueprint setup we can also easily integrate more trivia api's into the game down the road.

Immediate Development Goals

While we are waiting for the PhonalAI tools to come out of alpha we plan to utilize the great Pocketsphinx plugin for UE to try and package a fully working version (with true/false questions only).  This will remove the current need to package with Python for speech processing, but greatly limits the phrases that can be recognized.  Once a better STT engine can be utilized natively in Unreal Engine we will re-add other question types to the game.  Stay tuned for updates on our progress and when the new version is available for download to try out.

We had a ton of fun pushing the limits of what Unreal Engine can currently do with voice, and hopefully our next release will be something that everyone can enjoy playing with the family.