Building a Flexible Vision Framework for Video Games

During my time working for a previous game studio I had the privilege of building a number of core gameplay systems from the ground up.  We had a need for a server authoritative vision system that would allow objects in the world to be in certain vision state such as hidden, spotted and fully visible.  If the object was hidden the server would stop networking it’s position and state data and the client would not render it.  This ensured that clients cannot hack the executable to display the enemy position.  I referred to this system as the Intel system.

Another developer had written the first version of this system and he successfully built it to meet the requirements set by design.  However, design later changed these requirements and made a request that the system was almost entirely incapable of meeting.  With much work another developer was able to make it meet those requirements.  However, at this point the systems performance was a serious concern.  To make matters worse, design requested another change.  A 3rd developer similarly wrangled that one in.  At this point the system was considered on it’s 3rd re-write.

My boss originally accepted the task of rewriting the system.  Within a week he realized he would never have time to get the work done and asked me to take it on.  I accepted and proceeded to dive into the design documents.  Over the following two weeks I put together an engineering proposal on how the system would be architected.  I learned what went right and wrong with the original system and while I scrapped the original system, one or two concepts (I’ve forgotten the specifics at this point) did come over.

The key requirements that I needed to consider were as follows:

1. Design will flip flop on the rules that determine if an object is visible.  Perhaps the objects become visible automatically within a certain distance, perhaps objects are visible if an ally can see it or maybe objects become invisible if the player is damaged by a certain type of attack.  Second
2. The system must be flexible in the types of queries it makes.  It should be easy to swap out a query for another one if performance becomes a problem.
3. Objects should become visible as soon as possible when the rules for being visible would return true.  This requires the system to be very performant and with this, clever in how it manages it’s queries.
4. Players should not have a random advantage over each other.  If one player processes Intel against another, the other must process Intel that frame as well.

System Architecture Ideas

The Intel system was a large and complex beast.  There were a number of moving parts, a large number of classes and components and an architecture for putting all of this together in a way that would maintain flexibility and minimize the performance cost was critical to it’s success.

The “Pipes” Concept

One of the pieces my boss originally recommended looking at was the logic that determined if an object is spotted.  He recommended viewing this logic as a bunch of pipes.  These pipes may have one entrance but each pipe splits into one or more additional pipes.  Each pipe has a set of requirements in order to enter it and a set of output data.  Perhaps the first pipe is distance based.  If the player is within a maximum spotting distance, then we will flow through to the next pipe.  That pipe may check if the player is very close and if so, immediately resolve to a spotted state.  If that pipe fails it may lead to a raycast pipe who performs it’s check and returns it’s results.  The next pipe processes those results and if it returns success, passes along to yet another pipe who does even more processing.  All of these pipes and requirements would be able to be customized by design.  This would allow them, in the tool, to redefine the flow of logic for determining the Intel state of an object.

I wrote a quick 2 day rough draft of this system to determine it’s feasibility.  I felt ok about it, but I had concerns about the flexibility they system provided and how much engineering time would need to be spent on it.  I wasn’t convinced that amount of flexibility was necessary and I was worried about designer changes creating bugs and performance issues.

I discussed this architectural proposal and my concerns at length with my boss and our tech director.  In the end they agreed to let me make the call.  I abandoned that direction.

My Resulting Concept

I preferred to simply hard-code the spotted logic.  I recognized that design will want changes to this code flow, but after writing the logic for their initial design as a rough draft it was no more than 100 lines of very easy to follow code.  If design wanted to add a new piece to that flow, it would be simple.  I also recognized that it was unlikely that design would want to make consistent changes to this flow due to design being very well vetted.  I was right.  Over the course of the next year design requested edits to that flow maybe 3 times.  Building out the pipe architecture would have been a waste of time, all at the expense of much more complicated code that would be vastly more difficult to debug.

 

Final Architecture

So, without further ado, my next steps were to define the remaining system architecture.  The spotted code discussed above was a very small piece of this pie.

Here is a very rough flow diagram of the system architecture:

Classes

• IntelMgr:  Responsible for maintaining all permutations of Intel objects in the world.
• IntelQueryGroup:  Responisible for maintaining all permutations of IntelQueries from object A to object B.
• IntelQuery:  Responsible for executing a single IntelQuery.  There are roughly 3 different types of queries:  Distance, raycast and occlusion.
• IntelComponent:  Responsible for processing the IntelQuery results and determining the Intel state of the target object.
• IntelStateComponent:  Maintains the objects current Intel state of all other Intel objects in the world.
• IntelStateCalculator:  Fed the current Intel state and determines the current visibility state.
• IntelVisionTargets:  Feeds the IntelQueryGroup to determine what designer specified points on the target object to raycast to.
• IntelViewports:  Feeds the IntelQueryGroup to determine what designer specified points on the source object to raycast from.
• IntelSettings:  Global settings provides settings that individual objects don’t need.  Individual objects contain overridden settings or custom ones of their own.  These settings feed the IntelStateComponent’s IntelQuery processing.

High Level Flow

IntelMgr is the root of everything.  The client and server both call Update() on this and it processes everything it needs to.  It is directly responsible for tracking the Intel based objects in the world.  Each object has an entry added into it’s priority queue for each other Intel object in the world.  During the update it will allow a certain “cost” of these pairs of objects to be processed.

These are processed through IntelQueryGroups.  This class maintains a lost of all possible IntelQueries object A could make against object B.  This includes raycasts from each viewport to each designer specified point on the other object.  A certain number of IntelQueries are then run against both objects.

An IntelQuery can be raycast, distance, occlusion, etc based queries.  An IntelQuery must run at least a few of these in order to fully process the query.

Next the IntelComponent’s Update() is called.  This processes the query results and runs the logic I discussed in the “My Resulting Concept” section above.  The resulting state of the queried object is stored in the IntelStateComponent.

The IntelStateComponent’s job is to track the current Intel state of the object.  However, it doesn’t do this on it’s own.  It relies on an IntelStateCalculator to track the interpolated state.  This allows objects to fade out over time or even linger before disappearing entirely.  Without that players would have a jarring experience with enemies popping in and out of view.

Server/Client Differences

The whole system can be thought of as a Model View Controller.  The client is the view, the Intel objects are the models and the server is the controller.  The server naturally processes the above-mentioned “High Level Flow”.  The IntelStateCalculator will call a supplied callback when the state changes.  This will send the updated Intel state down to the client.  The networking systems ask the IntelMgr if the object is visible to a particular client and if so, they will send the primary object update packet down, otherwise they do not.

The client has a number of differences to the above flow.  It is overall much simpler.  It is notified of Intel state changes and sets these on the IntelStateCalculator.  The calculator processes this state just like it does on the server.  The difference here is that the listener for the callback is instead UI or audio.  The IntelMgr simply ensures that all the IntelStateCalculators are updated.

Performance Considerations

IntelMgr

Since IntelMgr is the root of everything, it has the ability to have the greatest impact on performance.  If the game world has 20 Intel based objects on each team then IntelMgr has 400 permutations of IntelQueryGroups and each IntelQueryGroup could have as many as a dozen raycast targets.  It originally had on average 23!

IntelMgr therefore needs to be able to minimize how many IntelQueryGroup updates are ran.  It also needs to track the cost of the group running in case the IntelQueryGroup needs to make an expensive update.  IntelMgr was built on a priority queue to solve for this.  Intel object nearer to each other would update more often.

IntelQueryGroup

This class also turned out to be critical to the performance of the system.  It also saw the most change during it’s development and in the end, was a bit more messy than I would have liked it to be.  It was turned into a threaded operation in order to provide more throughput and leverage our idle physics threads.  Additional complexities were added to minimize the number of IntelQueries it ran.  It was discovered late in development that the time that an object would become invisible again after being spotted was variable.  In order to resolve this I split it up into 3 phases:

1. Target is invisible:  Round robin through IntelQueries.
2. Target is visible:  Run only the last successful IntelQuery.
3. Target just lost:  Run a BatchedRaycastAll.

If the target is not currently visible, IntelQueryGroup would run a few IntelQueries per frame, not all of them, but it would run through it’s own internal priority queue of IntelQueries so that higher priority (more likely points to spot) are ran more often.  Once the target is spotted, the successful IntelQuery is stored off and each frame after that is the only query tested against.  This allows performance to be dramatically less for visible targets.  However, in order to ensure that targets fade out after a consistent time when they become obscured by the world I ran a BatchedRaycastAll the frame the stored IntelQuery failed.  This query would run all possible queries against the target.  If successful, it would cache off the first successful query in the batch for the following frames.  Naturally this BatchedRaycastAll is quite expensive.  It too was multi-threaded and we managed to take it down to a very manageable cost.  IntelMgr will receive information from IntelQueryGroup so it can determine how many more queries it can run that frame.

IntelQueries

These encompass a set of derived classes.  A query by itself just as a success flag to allow IntelComponent to process it’s logic properly.  I designed this to allow us to develop new queries easily and swap out queries that we felt weren’t performant or to allow us to try new query concepts.  While we stuck with distance, multi-threaded raycasts and occlusion queries we discussed other high level rejection queries such as bounding volumes, voxel data and Umbra.  Each of these have possible application here that could have allow trivial rejection of the remaining queries, saving time on raycasts.

Results

The system took approximately 4 months to build and 4 months to feed design requests for additional changes.  These changes proved easy to add in.  Adding multi-threading support took myself and another developer about 2-3 weeks of time.  The system worked well from the start.  The first check in proved to be bug free according to QA and as far as my own heavy testing of it found.  It proved to meet designs requirements, be performant, scale-able and easy to maintain.

Key Takeaways

• Adding multi-threading support was much more challenging than expected.  This wasn’t multi-threading itself, but the desire to leverage that to provide guaranteed fade out timings for Intel objects.  This code was starting to become challenging to follow and contained a few bugs that were difficult to track down.
• Needed more focused QA passes.  Much of the QA passes that were done were individuals on a single machine rather than a full server with all clients.  An 11th hour bug due to the multi-threading support was found and while relatively easy to fix, would have been caught weeks before with a proper QA pass.
• Testing the system thoroughly is relatively time consuming for an engineer.  However, it’s also very important to ensure bugs are not introduced.  I setup a test level that allowed me to run through some extreme examples to see the worst of the system and this helped speed this process up.
• Debugging state transitions through IntelStateComponent and IntelStateCalculator was tricky.  Finding a way to improve this interaction would have been a benefit.  I did send down the server state information to the client for debug rendering and this did greatly help.
• Don’t let design define how many IntelQueryGroups to execute on a given frame.  I originally put this in to allow them to tune the numbers until they got the responsiveness they wanted.  This caused our frame times to spike to 10ms from what should have been 2.  By adding in the BatchedRaycastAll query, this reduced the need for them to concern themselves with responsiveness.
• Be cautious about the number of IntelVisionTargets design can specify.  Lay out serious ground rules and ensure they stick to them.  Our original IntelQueryGroups had 23 raycasts, we reduced this to roughly 8 with a few exceptions allowing for more.  Doing this increased responsiveness for initially spotting a target and allowed IntelMgr to run more IntelQueryGroups per frame.
  • The goal had been to allow strangely shaped world geometry to ensure that an Intel object would still be spotted.  However, in the end our game found this to not be a concern.
• Keep accurate documentation.  This includes engineering and design “how-to” documentation.  This made it easy for other engineers to understand the system and was a handy reference for myself.  The design facing documentation proved to be extremely handy in allowing design to work with the system without needing to ask an engineer.