Performance Issues of UT Gameservers


I read a couple of papers and Internetpages I found to figure out whats the best configuration of an UT Server and what parameters influence its performance in what way.

Here is what I found out. Most stuff is pretty obvious, I guess you allready know that.

There are mainly 2 Issues:


A. Bandwidth of server / client connection
B. CPU Power of the given server



Both Aspekts are discussed in Detail in the following Sections. please note, that this Document is about UT Servers, and not about UT2003. Anyhow, many aspects of this are very much the same with UT2003/4, since UT2003/4 uses the same engine then UT, only a much newer Version. Most Aspects are also true for Games which are based on the Unreal Engine, such as Splinter Cell, Star Trek Elite Force, Tactical Ops, Strike Force and many others.



For your information:

Unreal 1 -> Engine Versions up to 250 (226 is the last patch I guess)

UT -> Engine Versions 300 up to (currently) 451

UT XBOX (aka Unreal Champinoship)- > Engine Versions up to 900

Unreal 2 -(aka The Awakening)> Engine Version 1400 (current 1403)

UT2003 -> Engine Versions around 2200 (currently the highest version is 2225.1)

UT2004 -> Engine Versions 3000 and up (the demo has 3120)

Unreal 2 XMP (multiplayer Version of Unreal2) -> Engine Numbers around 6400 to curently 7710 (these numbers are a bit out of order, since the XMP is more a patched 1400 engine then realy an enigne later then the UT2004 engine , this might lead to a number conflict later...)

A. Bandwidth Issues

What Bandwidth does a UT-Server need?

That depends on how many Players want to connect at a given time and what Bandwidth all these players need. The players bandwidth is dependent on 4 Parameters:
1. what connection bandwidth does a player have?
2. what is the max connection speed configured in the players .INI File
3. what is the max connection speed for a single player configured in the .INI File of the server
4. what is the max throughtput connection speed of the path between the player and the server

All of these issues are discussed below:
First of all: As a general rule of thumb the LOWEST bandwidth of 1-4 determines the players connection speed to the server. Within these topics lays a lot of reasons for poor gameplay.



1. what connection bandwidth does a player have?


This is the raw bandwith of a players connection, thus how big is his line. This may vary from 56 K modem lines to XDSL or cable lines. XDSL lines come in various flavors. A main difference is syncronictiy. ADSL Lines (thus ASYNCRON DSL) have a good DOWNSTREAM bandwidth (from the net to the player) but a small UPSTREAM bandwidth (from the player to the net). SDSL Lines are syncronus DSL lines, both direktions have the same bandwidth.
The vast amjority of DSL lines is asyncronus. The have as an example 700K downstream, but only 160 K upstream. In such a situation the player could only send 160 Kbit/s data to the server and Receive up to 700 Kbit/s.

This fits quite well the players needs:
Allthough mostly UT Traffik is syncronus, it usualy takes never more bandwith then about 100kbit/s (and even that only under rare circumstances, such as Exesive Servers). So most UP channels are broad enough for gameplay, and the broader DOWNlinks are good for decent map download.

That is also the reason why 56K Modem lines are worse then they should be: a 56K modem does some sort of switching in the upstream and downstream to get the maximum bandwidth and also needs to convert signal from digital to analg and back, this switching and processing time causes a bad ping. A 64Kbit/s ISDN line, allthough only 20% better bandwith gives a much smother gamplay.

So for most situations not the limited bandwidth of a modem itself is the problem, but the dataprocessing which results in bad ping. (ping is discussed more closely in 4. of this chapter)

Unfortuantaly this is not the onyl item in players connection. Another issue is the type of error-correction the line-provider builds in his lines.
Many providers do errorcorrection on linelevel (OSI-Layer 2). Thus any package which is uncomplete, has checksum errors or other problems is discarted. The line-terminators, means the boxes which are installed at each lines end, most people call it MODEM or ROUTER (which is both wrong) communicate with each other to RESEND the discarted faulty package.

This gives the customer the illusion of an error free line, all data pakages which are coming out of the line-terminator box are 100% error free. The backdraft is, that some of them are coming a bit later then they should... namely the resent faulty packages.

But also all pakages which are okay will travel a bit longer through these lines due to the extended checking process. This gives higher lantency to such lines. Please note that "latency" is the technical term for the UT-Term "ping". So whenever a UT Gamer says he has "bad ping", a network technichan whould say "the conection has high latency".

Some providers don´t do error-corection on Layer 2 (the connection layer) at all. Here the faulty packages are presented to the computer and the application which needs the packages... in our case UT. UT itself has then to figure out faulty datapakages and deal with them.

Unfortunately, UT is pretty good in dealing with faulty pakages up to a certain percentage (about 2%-3%), but VERY BAD in dealing with datapakages which are coming later then they should. (if you think about it... pretty logical, because timely presentation of a given persons position is pretty essential for UT)

As a result all players who have a line WITH built in error-correction will have gameplay with much higher ping then players with lines which do NOT have error-correction!

Example: Germans T-Online T-DSL Lines are with error-correction. All players using this lines are encountering laggier gameplay in comparison to players with ARCOR lines, which do not have error-correction on line level. (Marketing whise ARCOR calls the lack of correction on line level "fast forward mode") allthoug the raw bandwidth of both lines might be the same!



2. what is the max connection speed is configured in the players .INI File


This is the part in the .INI File where it reads:

[Engine.Player]
ConfiguredInternetSpeed=20000

This number may be configured in the OPTION menue by choosing the Linetype such as LAN, Cable, modem etc. To make things a bit more complicated this gives the desired bandwidth in BYTES per Seconds, not bits per second.
To calculate the corosponding line bandwidth, you need to multiply this with 10 (8 bits per Byte plus start and stop bit). To calculate the numbers needed in the .INI file from the line speed you have, simply divide it by 10.

Example: You have a 56 Kbit Modem, which actuall connects with 44 Kbit/s. That is 44*1024 (becaus as you all know 1 K is in computer land 1024 not 1000) makes 45056. Diveded by 10 makes some 4500. So thats the number you should type into your .INI File.

If you type a larger number then your line can actually handle, you will increase packet loss, because the server sends you more data then your line can deliver. BAD thing. So be carefull with this. It is very important to match this number with yout downstream rate. If in doubt, make it LOWER.
It makes no sense to get higher then 25000 here, resulting in 244 Kbit/s. So even if your connection can handle more then 244 Kbit/s that does not help you, because more data will be not sent from the server (well that is not EXACTELY true, as discussed later).

Most sources say, that u will rarely have more 8000 bytes/sec comunication with a server. You can check this yourself by hitting F6 during gameplay and watsch the numbers. U will almost never see somthing above 8000.

So if you are looking for a new line for your home gaming, more then a 256 Kbit/s line SDSL (Syncrone) is not needed for very decent UT connections! Even a 128 kbit/s line is absolutely suffient.

Anyhow, this does not determine the connetion speed alone:



3. what is the max connection speed for a single player configured in the .INI File of the server

The server has in its .INI file a very simmilar section:

[IpDrv.TcpNetDriver]
...
MaxClientRate=20000

This is corrosponding to the bandwidth settings in the players .INI File. If the bandwidth on both sides are different the client connects with the LOWER speed of these 2 settings. Many servers still have the default here, which is 5000 ONLY! So even if you have a Super-UltraDSL Line with 455 TERRAbit Bandwidth and configure your .INI File to 23534636, you will still geht 5000 as configured on the server.... thus onyl litlle more then a modemuser!!!

Rising the MaxClientRate on the server side gives 2 problems. First and obvoius it eats up the servers conection bandwidth rapidely. Lets assume, the server is behind a 2 mbit/s S2M Digital line (pretty good in the world of internet). If you grant per MaxClientRate=5000 every client only maximum aprxiamatly 50 Kbit/s (a very good modem line) then the Server can be played by some 35-38 players with no problem. Since the maxium a UT Server can have is 32, you then should never have problems resulting from BANDWITH.

If you rise MaxClientRate to 20000, then that means that every player can eat up to 204 Kbit/s (if their line is good enouth). As a result onyl 8 players can connect to the server! (well... only if the cleint really eats up the bandwith. Thats usually only possible in extremyl rare situations such as Mr.Pants Exessive servers or the like)

B. Server CPU Power issues



Generall CPU Issues



Most CPU Power in playing UT is needed to calculate the actuall picture. So the clients computer should have a good display subsystem ("Graphic Card") as we all know. I´m not going to discuss these matters here... only one thing maybe: The most important factor is how long the computers display subsystem needs to calculate the poygone structure which ends up to the inplay picture.
This then results in the max framerate. The unreal engine tries to maximize framerates through dynamicaly lower the quality if the MaxDesired Framerate cannot be reached with the current quality settings. Usually a framerate of about 30-40 is enouth, more gives IMO no better gameplay. Almost all computer today can get that. Nevertheless at some occasions gameplay stutters. This usally happens when the polygones in sight are risign above a certain level. To minimize the polygones UT has to handle at a given time, maps are divied into "Zones", so that the UT engine has only to take all polygones in this zone into consideration. If some of you have worked with UnrealEd, you know the concept of Zones: It divides the map into subareas.
Some maps do have very large zones, especailly large open land maps, because a zone must minimunm reach that far as a palyer can look in the map. These maps tend to stutter on slower machines, some of them have so large zones, that even my Macintosh cannot handle them (har har har).

Please note also that here lies the main reason for the UTbuild2000+ (known as UT 2003) performance issues: this engine can handle maps with 100+ more details then the UTuild300-500 (UnrealTournament) engine.

The server however does not need to display anything. So even slower machines can make decend servers. To have a look on the servers CPU needs, lets first have a look on what the server has to do:
The server keeps all information of the world as it is in geamplay. The server knows where all actors are at a given point. Please note, that the term "actor" refers in UT not only to players but to EVERYTHING which can move in any way... so basically everything beside pure terrain.
Missiles are actors. PowerUps are actors. Weapons are actors. Smoketrails are actors. Doors are actors.

Think of the server as a huge EXCEL-Sheet. In the cells are some numbers for ALL actors in the current game. The numbers determine postion, speed, acelleration, direction, time and the like.
All cells have formulars attached which are able to calculate how the actors numbers change trough time. If you fire a rocket in this direction, the Rocket ist an actor. Inside the large "Excelsheet" of the server is some cell containig data as: ROCKET, Fired by, fired at, fired when, has this speed, heads this direction, has this time to life befor exploding etc.
UT now recalculates this "Excelsheet" from time to time. All these results are sent to the players computer if he needs them, thus if he can SEE the given actor. So if you can see the rocket, the server knows that by comparing the actor ROCKET and the actor PLAYER in his "Excelsheet". If the are close by and nothing is in the line of sight, then they can see each other. If that is the case, the server sends the information via Internet to the players UT engine: "Hey fellow, please show your Computerowner a rocket, heading this and that direction. New information will follow shortly..."

Obvioulsy the more often the server can calculate his "Excelsheet" the smother is the gaemplay. Imagine, the server could do that only one time per second, then the rocket whould not change its position for one second on your screen. Please note, that it whould not tracel in slow motion, the server automatically takes the passed time into calculation. So the rocket whould after one second "jump" on the postion it should be one second later.

The UT Server usually recalculates his datasheet 20 times per second. Now this is interisting in many ways. Lets have a view on some effects all this might have:

The calcualtion of the servers "Datasheet" needs more CPU Power the more actors you have in the game. So as well larger maps with more Powerups, Waepons, doors cars etc as more players need more CPU Power in the server. In oposite to the client, Zones or the like do not help, the server needs to know the state of all actors all time. If at a given point the server cannot handle the calcutlions any more he drops cycles. Lets assume, the given server needs 1/18 th of a second to do a calcultion cycle, then the server cannot calculate 20 cycles in the second anymore but only 18. This may vary inside the game pretty much, because the number of actors changes a lot, depending on the number of missiles shot. This is esentially critical in games where many missiles are on its way, such as PANTS EXSESSIVE Games.

To eleminate the effekt this whould have on gameplay, the client tries to prepredict what happens in the meantime. If the clients UT Engine does not get information about the travel of an actor it calculates ITSELF based on older information where the actor should be now (for example 1/2 second later)

This works very nice for rockets and missiles in general, because most of them do not change traveldirection after they have been shot. But for players or guided Redeemer Warheads (as an example) that is not true.

Thus if you see a player suddenly disappear right in front of your eys it might be, that a dropped connection OR a dropped calculation cycle was covered by your Cleint machine... only to learn 1/3 second later, that the player is NOT there, because he has turned to the left or the like.

Another effekt you all kown is that players "floate" in a given direction untill they hit a wall, this might also be a hint for Server CPU overload.

This leads us to a very important conclusion: Everytime the picture stutters, or the play gets laggy or the framerate drops, it is NOT the Servers CPU power, but either the clients graphic subsystem power or certain problems with the connection. Lack of CPU Power usually results in "jumpy" behavior of OTHER Players. Compare that to a totally connection loss. You all have noticed, that that does NOT results in stuttering images or things like that, only all movement off ALL actors beside yourself dont get any ACCELERATION anymore. You cannot shot, others either float in the direction they where heading last or the just stay where they are. Still you can run through the map with no problem.

Another thought: If the Server recalculates the scene "only" 20 times a second, whats the benefit of framerates higher then 20 on the clientside? Well, your OWN movement looks smoother till a level of up to some 30-max40 frames. Beyond that its absolutely useless.

Even worse: the Server calcualtes the scene all 1/20 second, but if he can send the results depends on the connection rate of the player! Modem players might get updates onyl 10 times a second or so!

Beside raw CPU Power to recalulate the scene, there are other needs the server has. Sending the appropriate data to lets say 20 Players is not an easy task. There are computers which cannot handle more than 1-2 Mbit/s in its network subsystem. Such a computer whould give a strange picture: You might connect it with Ethernet to the net. It might have a FAST CPU, which needs only 30% of its power to recalculate the scenes, all .INI stuff is confgured ok... nevertheless, only 10 DSL users can connect without getting lousy ping and paketloss. Wat´s wrong? Well the Server can calculate everything but his IP Driver might be to slow to deliver all the data to the Etherntadapter. The server has some sort of an INTERNAL bandwidth problem.

So raw CPU Power does not mean that the computer is a good UT Server, Mhz alone does not help. Allthough I have no good expirience in this I prepredict, that some old 500 mhz systems might make better servers then some 1.6 Ghz machines which are hardwareside illconfigured.

One last aspect : The Number of Cycles the server calcualtes per second can be changed in the server .INI File:

[IpDrv.TcpNetDriver]
...
NetServerMaxTickRate=20



Yep! So in this case a server does not recalculate the scene more then 20 times a second even if the CPU has lots of power left.

Obvieously, rising this level to say 30 WHOULD have benefit if:
- the servers CPU power is sufficient
- the server networking subsystems can handle the amount of data
- the servers Inernet connection can handle the amount of data
- the clients line can handel his part of the data
- the clients computer can handle the additional data
- the clients graphic subsystem can show more then 20 frames/second.

Rising the level to more then 30 is useless, because the human eye has a max resolution of somwhere between 25-30 frames per second (movies have 24 frames as you know).

Please note, that fumbeling with this has linear effekt on all components: 20 ticks versus 30 ticks means 33% more CPU load, 33% more bandwidth consumption etc etc.

Whew! Lots of reading to be sure. Even the bandwidth stuff where he includes modems, though you may think it out of date, I still think it is just as relevant. In these modern times we have throttling, lousy routing, sharing our bandwidth with neighbors, etc.

Hopefully the players can take something away from this.

Me as a server admin, I see two things I could play with. One I WILL play with, the other I am tempted to but will leave alone.

I am tempted to play with the tick rate (NetServerMaxTickRate). I've read many posts in the past about this setting, where people were trying to decide the best number to increase it to. I am tempted to decrease it. This would result in less CPU time needed, and less info the server would have to push to the players. But this may introduce noticable stutter. So nah, I'll leave it alone.

I think I will play with

[IpDrv.TcpNetDriver]
MaxClientRate

I'm going to reduce this number. This will also send less information to the players, but it culls the information in a different way, so I'm ok with this.

I think I have it, but send it to me also Herm!

As far as tick rate - I would say to leave it!
The characteristics of Some weapons change if this is changed.
From what I read from some experts, 20 is optimum for most all servers.
Linux servers MAY be upped very slightly (21 or 22)? to compensate for some slower aspects of its tick, but I think GameServers already does this.
Other tick rates than 20 is Not recommended.

I sent the .mht to HOOK. I asked him to send me your email since I was too lazy to look it up on my forum. Maybe I attach it this you now. Here you go!

No thoughts on this Helen?

I did read it. Alllllllll of it. LOL. What don't you agree with?

Well said. I agree. this game is very old now and connections sucked back then. I'm sure there is plenty of room for tweaking. Still though, I have found stickying toabout the number 3000 instead of 5000 makes everything work pretty well and maximizes player slots for servers with linited upload speed. When I found this and started CG my maxupload was 360. Not 1,500 like it is now. I was playing on the PC that was serving too. A listening server.