Game Flow: Distributed UI
While I was developing Re-Spawn (not that I have stopped, but that is beside the point) I started coming across various needs for the game that were not necessarily covered by the game engine itself. Things like master servers, license keys, achievements, and other standard game features; obviously the game engine is there to help you make a game and I wouldn't expect the game engine to provide me with these and so I started working on my own services that would eventually become Game Flow.
Game Flow is a technology stack that provides game developers with functions and features that include but are not limited to the ones listed below:
* Scalable, Robust, and Lightweight Master Server
* Scalable, Robust, and Lightweight Game Update Server
* Scalable, Robust, and Lightweight Achievements and Statistics Server
* Secure Encrypted Communication between the Interconnected Services
* Secure Server License for Official Servers
* Secure Player Profile Storage on Client Machines
* Secure Player Storage and Allocation of Achievements and Statistics
* Web Based Server Control Panel (Game Changer)
* Web Based Game GUI that is Theme-able and Supports Internationalisation (Ditributed UI)
* and many more...
I will cover all these topics in due course but for today I want to focus on the Distributed UI system and how it works.
How did the concept come about?
In all the prototypes I have developed and all the iterations of Re-Spawn that I have done, I have also wondered how would you go about creating a GUI System that allowed you to internationalise it if you had to. I took various different approaches and they all the same fundamental problem, the logic sate with the game client and required extensive scripting and/or modifications to the game engine and so I abandoned these ideas.
While taking a break from game development one day and playing some Guild Wars 2, I noticed that the game GUI for the game seemed to be remotely accessed from the game client or was being streamed some how and that is when the idea hit me ... why not take the game GUI out of the game and make it work off a separate remotely accessed server. Armed with this idea I started researching various technologies and tried a few different things, which didn't work out as expected and I started getting a little bit demotivated.
It was around about this time that Stefan Lundmark released an update to the T3D Awesomium Integration Pack and I decided to buy it, for an entirely different purpose. Basically I wanted the players in Re-Spawn to be able to go to a Computer Screen in the game and be able to pull up their statistics from a website and that is when it dawned on me, Stefan had come up with the solution to my problem: How do you integrate the website with Torque 3D?
I integrated Stefan's pack in to Re-Spawn, created a test web-application in Java and JSP and linked the two and I was completely blown away by the result. The screen shots below show my two initial screen shots, the first one is the login screen running on the web server and the second screen shot is the same login screen running in the game. This is how the Distributed UI System came about.
What technology was used?
I have been developing in Java since late 1997 and over the last 8 years or so I have been developing Web Based Application using JEE and JSP, so it was a natural choice to choose the language that I was the most familiar with and that I had a good track record with. I personally developed an application that handles over 30 000 page-views a day, 100 concurrent users, and serves these pages in less than 500 milliseconds ... using 1 server.
The next choice I mad was to use Linux as the main server, this is primarily due to my experience with and continue use of Linux since early 1996, and add to that the FREE cost of licensing fees and low resource usage for server, it was an obvious choice.
Next I chose PostgreSQL as the initial database but after a few initial tests I switched over to MySQL for the initial development. I have however implemented Hibernate and I am using an Entity Framework in conjunction with HQL (Hibernate Query Language) which means I can rapidly switch to any Hibernate supported database should the need arise.
The final choice that I made was to use Apache Tomcat as the JEE application server, it is more lightweight than Glassfish but provides me with the extra features that I need in order to handle high loads that might be encountered. It also integrates quite easily with HAProxy and the Apache Web Server to ensure that I have a decent load balancer in front of the systems.
There were also extra technologies that were chosen that included things like Redis (Caching Server), Spring Framework (JEE MVC Platform), Apache Velocity, Apache Tiles, and a few more that were essential to creating a robust, scalable, and stable system. The last thing you want is the server to fail while you have players connecting to it, the diagram below shows how the architecture fits together and where the various services reside.
How long was the development?
From start to finish, the development took approximately 6 weeks of after hours and weekend work to complete. During the entire time I focussed on the fundamental features that I needed to support in order to make the service work, these were mainly:
* Internationalisation
* Secure Sessions
* Fast Response Times
* Slick User Interface
* Consistent User Interface
* Easy-to-use Interface
The User Interface requirements were met by using jQuery and Bootstrap 3, that were customised and modified to match the look and feel of the game in question. Overall I am very happy with the results achieved and the screen-shots show what I achieved with the Distributed UI system.
How scalable are you?
Currently the Game Flow system is running off 1x Linux server in Denmark using a cloud hosting company called Digital Ocean. With my current Game Flow Master Control Panel, I am able to bring up extra services in the various supported regions with a click of a button. The game client is designed to connect to the nearest regional service during start-up using the HAProxy system and some other features that I wrote.
These extra regional servers provide an added bonus of the Distributed UI system automatically being displayed in the regional language if applicable, this can be overridden by the user and the Game Flow Master Control Panel if the need arises.
Add the MySQL Clustering technology, HAProxy Load Balancing and the relative low cost and ease of bringing up extra servers within a matter of minutes ... automatically ... I feel that Game Flow is extremely robust and scalable, but as with anything only time and volume of traffic will tell. I am however silently confident that it will stand up to the load requirements.
How secure is the service?
This is always a tricky question and you want to answer it as carefully as possible without giving too much away but I will state the following:
* Passwords are stored using a one way encryption and there is no way to extract a decrypted version of the password from the database
* Passwords are encrypted with a salt value
* Communication between the game client and the Distributed UI systems uses SSL (i.e. HTTPS) to prevent middle-man attacks and spoofing
* User sessions are controlled by the Spring Framework using their Spring Security features
* Critical information between the game client and server are encrypted using RSA Public Private Key encryption
* The game client only has access to its public key and the server keeps the private key
* Wherever it is required, magic tokens are used to prevent data corruption
* Finally we use SQL Injection Prevention Techniques to handle any surprises that we may not have thought about
Ultimately the system is as secure as we can get it and we are continually monitoring for suspicious activity. All actions are logged in a log file and we can draw activity reports from this to spot any potential issues that may arise. All the systems are backed up regularly and we do not store any critical information in the database that could be compromised (like personal information or financial information), we prefer to use third party providers for this information as this is what there core business is.
Where to from here?
Well the next Alpha Release of Re-Spawn is due for the 1st of April 2014 and it will be launched using the Game Flow technology, with some minor features disabled at first. This will be a nice "live" test of the Game Flow system and will give us some valuable information while the testing is going on. Once we are happy with the results and are confident that the service works as expected we will start rolling out the service to our other projects that are currently in Prototype or Alpha state.
Initially we wrote the software and services to address requirements that we had for Re-Spawn but we may start looking at the possibility of selling licenses for this technology to other interested parties, please don't quote me on this as I do not have a definitive answer on this yet and will not be able to give a definitive answer for a while yet.
In the next update, I will talk about the Master Server.
Gobbo Games Marketing:
Website
Facebook
Google+ (WIP)
Twitter
Skype: gobbogamessa
G-Talk: webmaster@gobbogames.co
E-mail: [email]webmaster@gobbogames.co[/email]
Re-Spawn Marketing:
Website (WIP)
Forums
Blogs
Facebook
Google+
Twitter
Skype: ggrespawn
G-Talk: respawn@gobbogames.co
E-mail: [email]respawn@gobbogames.co[/email]
Technology Links:
Apache Web Server
Apache Tomcat
Apache Tiles
Apache Velocity
HAProxy
Hibernate
MySQL
Spring Framework
Game Flow is a technology stack that provides game developers with functions and features that include but are not limited to the ones listed below:
* Scalable, Robust, and Lightweight Master Server
* Scalable, Robust, and Lightweight Game Update Server
* Scalable, Robust, and Lightweight Achievements and Statistics Server
* Secure Encrypted Communication between the Interconnected Services
* Secure Server License for Official Servers
* Secure Player Profile Storage on Client Machines
* Secure Player Storage and Allocation of Achievements and Statistics
* Web Based Server Control Panel (Game Changer)
* Web Based Game GUI that is Theme-able and Supports Internationalisation (Ditributed UI)
* and many more...
I will cover all these topics in due course but for today I want to focus on the Distributed UI system and how it works.
How did the concept come about?
In all the prototypes I have developed and all the iterations of Re-Spawn that I have done, I have also wondered how would you go about creating a GUI System that allowed you to internationalise it if you had to. I took various different approaches and they all the same fundamental problem, the logic sate with the game client and required extensive scripting and/or modifications to the game engine and so I abandoned these ideas.
While taking a break from game development one day and playing some Guild Wars 2, I noticed that the game GUI for the game seemed to be remotely accessed from the game client or was being streamed some how and that is when the idea hit me ... why not take the game GUI out of the game and make it work off a separate remotely accessed server. Armed with this idea I started researching various technologies and tried a few different things, which didn't work out as expected and I started getting a little bit demotivated.
It was around about this time that Stefan Lundmark released an update to the T3D Awesomium Integration Pack and I decided to buy it, for an entirely different purpose. Basically I wanted the players in Re-Spawn to be able to go to a Computer Screen in the game and be able to pull up their statistics from a website and that is when it dawned on me, Stefan had come up with the solution to my problem: How do you integrate the website with Torque 3D?
I integrated Stefan's pack in to Re-Spawn, created a test web-application in Java and JSP and linked the two and I was completely blown away by the result. The screen shots below show my two initial screen shots, the first one is the login screen running on the web server and the second screen shot is the same login screen running in the game. This is how the Distributed UI System came about.
What technology was used?
I have been developing in Java since late 1997 and over the last 8 years or so I have been developing Web Based Application using JEE and JSP, so it was a natural choice to choose the language that I was the most familiar with and that I had a good track record with. I personally developed an application that handles over 30 000 page-views a day, 100 concurrent users, and serves these pages in less than 500 milliseconds ... using 1 server.
The next choice I mad was to use Linux as the main server, this is primarily due to my experience with and continue use of Linux since early 1996, and add to that the FREE cost of licensing fees and low resource usage for server, it was an obvious choice.
Next I chose PostgreSQL as the initial database but after a few initial tests I switched over to MySQL for the initial development. I have however implemented Hibernate and I am using an Entity Framework in conjunction with HQL (Hibernate Query Language) which means I can rapidly switch to any Hibernate supported database should the need arise.
The final choice that I made was to use Apache Tomcat as the JEE application server, it is more lightweight than Glassfish but provides me with the extra features that I need in order to handle high loads that might be encountered. It also integrates quite easily with HAProxy and the Apache Web Server to ensure that I have a decent load balancer in front of the systems.
There were also extra technologies that were chosen that included things like Redis (Caching Server), Spring Framework (JEE MVC Platform), Apache Velocity, Apache Tiles, and a few more that were essential to creating a robust, scalable, and stable system. The last thing you want is the server to fail while you have players connecting to it, the diagram below shows how the architecture fits together and where the various services reside.
How long was the development?
From start to finish, the development took approximately 6 weeks of after hours and weekend work to complete. During the entire time I focussed on the fundamental features that I needed to support in order to make the service work, these were mainly:
* Internationalisation
* Secure Sessions
* Fast Response Times
* Slick User Interface
* Consistent User Interface
* Easy-to-use Interface
The User Interface requirements were met by using jQuery and Bootstrap 3, that were customised and modified to match the look and feel of the game in question. Overall I am very happy with the results achieved and the screen-shots show what I achieved with the Distributed UI system.
How scalable are you?
Currently the Game Flow system is running off 1x Linux server in Denmark using a cloud hosting company called Digital Ocean. With my current Game Flow Master Control Panel, I am able to bring up extra services in the various supported regions with a click of a button. The game client is designed to connect to the nearest regional service during start-up using the HAProxy system and some other features that I wrote.
These extra regional servers provide an added bonus of the Distributed UI system automatically being displayed in the regional language if applicable, this can be overridden by the user and the Game Flow Master Control Panel if the need arises.
Add the MySQL Clustering technology, HAProxy Load Balancing and the relative low cost and ease of bringing up extra servers within a matter of minutes ... automatically ... I feel that Game Flow is extremely robust and scalable, but as with anything only time and volume of traffic will tell. I am however silently confident that it will stand up to the load requirements.
How secure is the service?
This is always a tricky question and you want to answer it as carefully as possible without giving too much away but I will state the following:
* Passwords are stored using a one way encryption and there is no way to extract a decrypted version of the password from the database
* Passwords are encrypted with a salt value
* Communication between the game client and the Distributed UI systems uses SSL (i.e. HTTPS) to prevent middle-man attacks and spoofing
* User sessions are controlled by the Spring Framework using their Spring Security features
* Critical information between the game client and server are encrypted using RSA Public Private Key encryption
* The game client only has access to its public key and the server keeps the private key
* Wherever it is required, magic tokens are used to prevent data corruption
* Finally we use SQL Injection Prevention Techniques to handle any surprises that we may not have thought about
Ultimately the system is as secure as we can get it and we are continually monitoring for suspicious activity. All actions are logged in a log file and we can draw activity reports from this to spot any potential issues that may arise. All the systems are backed up regularly and we do not store any critical information in the database that could be compromised (like personal information or financial information), we prefer to use third party providers for this information as this is what there core business is.
Where to from here?
Well the next Alpha Release of Re-Spawn is due for the 1st of April 2014 and it will be launched using the Game Flow technology, with some minor features disabled at first. This will be a nice "live" test of the Game Flow system and will give us some valuable information while the testing is going on. Once we are happy with the results and are confident that the service works as expected we will start rolling out the service to our other projects that are currently in Prototype or Alpha state.
Initially we wrote the software and services to address requirements that we had for Re-Spawn but we may start looking at the possibility of selling licenses for this technology to other interested parties, please don't quote me on this as I do not have a definitive answer on this yet and will not be able to give a definitive answer for a while yet.
In the next update, I will talk about the Master Server.
Gobbo Games Marketing:
Website
Google+ (WIP)
Skype: gobbogamessa
G-Talk: webmaster@gobbogames.co
E-mail: [email]webmaster@gobbogames.co[/email]
Re-Spawn Marketing:
Website (WIP)
Forums
Blogs
Google+
Skype: ggrespawn
G-Talk: respawn@gobbogames.co
E-mail: [email]respawn@gobbogames.co[/email]
Technology Links:
Apache Web Server
Apache Tomcat
Apache Tiles
Apache Velocity
HAProxy
Hibernate
MySQL
Spring Framework
Comments
My own personal ideal though is to never ever create an online requirement unless absolutely needed, i.e. for a multiplayer-only game (maybe not even then). The idea of ownership/collectibility is very important to me.
You are able to host a local server but if it is not an official server you will not be able to perform any achievement unlocks.
After I finished the last few issues last night I ran a play test and scheduled screenshots to fire off every 15 seconds ... the gallery is the result of two very brief play sessions (will do a much longer one this weekend) -- https://www.facebook.com/media/set/?set=a.773615492649316.1073741830.353075714703298&type=1
I have a few more things to finalise before the next Alpha Release ... but last night was the completion of a major hurdle and cleaned the Game Play up to a stage where I am very happy with the result.
The video and the screenshots are still the original Factory map (re-textured and modified version of Q2DM8) and I will add more screenshots and videos of other maps when I test them.
My next update will talk about the Stats Tracking System that I have in Game Flow now ... oh yes ... we are tracking almost everything. Update will come through next week some time. 8-}