Jump to content

Raw UDP Sockets - Server Stability Note for Server Admins


Drathek

Recommended Posts

On 6.10.2017 at 6:28 PM, MaeL said:

Hi,

this is not really the subject, but this may be caused by their ""fixes"", have you any stability problem caused by the auto-decay of abandoned structures? I've noticed multiple server crashs and huge memory leaks (up to 16GB instead of 7/8) since last weeks caused by the auto-destruction of abandoned structures (much more than before), and a general deterioration of the game server perfs since official release; disconnecting, crashs, memory leak, freeze, rubberbanding 

Same thing with destroywilddinos command, heavier than ever, can cause 20 minuts+ freeze or even crash

-Linux
-TCP P2P
-TheIsland/TheCenter
-No Mods
-.ark files much smaller than a few months ago
-Dedicated server, no hardware problem/300mb stable connection
-Correct or native rates/parameters

 

...Anyone else?

- auto-destruction is executed when you restart the server, so this souldn't affect the servers performance while running.

- Wild dino wipes are the same as ever on The Island, The Center and Scorched Earth.  Only Ragnarok has more than 60 seconds freeze and therefore all players get kicked.  (If you have 20 minutes, then for sure something with your server is not ok)

Link to comment
Share on other sites

  • Replies 82
  • Created
  • Last Reply
4 hours ago, Toni said:

- auto-destruction is executed when you restart the server, so this souldn't affect the servers performance while running.

no, it may be executed on server restart if you havn't any player to trigger it, but most of the time its executed while running when someone load the area 
 

4 hours ago, Toni said:

- Wild dino wipes are the same as ever on The Island, The Center and Scorched Earth.  Only Ragnarok has more than 60 seconds freeze and therefore all players get kicked.  (If you have 20 minutes, then for sure something with your server is not ok)

ok thanks for your report I'll try some additionnal tests. BTW its seems that your servers are hosted on Windows not Linux, important detail

Link to comment
Share on other sites

Guest BubbaCrawfish
Just now, MaeL said:

BTW its seems that your servers are hosted on Windows not Linux, important detail

Not really when normal dino wipes only take a minute to occur. I'm on Linux, and if it locked the server up for more than a couple of minutes, I'd be seriously looking at what is causing that.

 

I run three servers on Linux, with the occasional fourth... Ragnarok only ever took a minute or so to complete.

Link to comment
Share on other sites

it's more like a delayed freeze, my weekly wild dinos clean was done this morning (programmed at 6h00 every wednesday) everything looks good on the moment, but there is always a server downtime after that, between 6h10 - 6h20

are you using cheat destroywilddinos or the new ?ServerAutoForceRespawnWildDinosInterval parameter?

Link to comment
Share on other sites

Guest BubbaCrawfish

destroywilddinos, but chances are if it's the new function you need to report it, and if you are simply using a script to do it, then you need to figure out what is going on.

 

This should not be happening in this manner either way.

 

 

Link to comment
Share on other sites

19 hours ago, MaeL said:

it's more like a delayed freeze, my weekly wild dinos clean was done this morning (programmed at 6h00 every wednesday) everything looks good on the moment, but there is always a server downtime after that, between 6h10 - 6h20

are you using cheat destroywilddinos or the new ?ServerAutoForceRespawnWildDinosInterval parameter?

Thats the reason why i don't use this wild dino wipe at restart function.  Changing the savefile while starting is really really risky.

For example if there would be a crash at startup (Like we had last weekend with Ragnarok), then its destroying your savefile because it is changing it at startup.

 

Here is the reason for the servers "downtime" / "no-response"

The problem is based on the save system of ark (Like i already reported 1000 times for other problems...And already posted a solution because Conan Exiles has a solution for it with the UE ! They key-word is SQLite Database ! ).

The freeze / non-response of the server is happening when the server is cleaning out the dinos out of the savestate (In RAM). The "database" system ARK uses, is completely trash. The server has too long to search / delete the entries in the savestate. 

Other problems like for example the freeze on Clientside which you get when you transfer a big amount of single items, is exactly based on the same thing.

 

For people which doesn't believe me (There are always such ^^ // Including wildcard), here is how to prove:

1. Run a wild dino wipe on a Ragnarok Server which has a 200+ MB save file --> Result: Server freezes for minimum a minute and kicks all players out

2. Run a wild dino wipe on an emtpy Ragnarok Map which only has the wild dinos and nothing other --> Wild dino wipe finished in a few seconds, no kick of players.

 

Other examples to prove the bad save system of ARK:

1. Run a server with a big savefile... but no players connected, only you for testing (So server fps is at max of 30 fps). 

2. Move many items from one container to another container (Have a look at how many items per seconds you can move)

3. Run a server with a very little savefile. Only place 1 storage box on the map and fill it. No players connected, only you for testing (Server fps at max of 30 fps)

4. Move many items from the container to another container  (Have a look at how many items per seconds you can move).

 

You will see that the empty savefile server is about 5 times faster than the big savefile server. Even if both has no players connected and the only difference is the savefile size.

The bigger the savefile... the bader the server performance by acessing something out of the save

 

More other examples:

It is reproducable with ALL ingame actions which needs to change something on the savesystem. For example remove big bases with command.

 

The SOLUTION for this problem (Listen Wildcard !! Please start to listen to me!!):

Rewrite the save system of ARK and use the SQLite possibilities of the UE !    (Or even better do it with MySQL ...so when can move the database away from our ARK servers).

The Conan Exiles devs (Using the Unreal Engine too) does use it. Result --> No laggs at server autosaves / No laggs at big item transfers  / No laggs when you remove many things from the map

It was simply wrong to program ARK (which has big save files) with this File-Based Save System  (The same for the Cluster directory).  Thats simply a way which things was programmed before 10 years.

Databases was invented because they are way faster (specially when it comes to large saves like ARK has)  than File based Save Systems like you use for ARK!

If you don't have the competence / capacities to rewrite your savesystem, simply hire one of the Conan Exiles Devs ! !

Yeah i know... this is months of work. But it would solve the most performance problems of ARK.   (Wouldn't it be nice to eliminate the autosave laggs/rubberbands all the servers out there (including your own) have???))

 

Link to comment
Share on other sites

Guest BubbaCrawfish

I thought that would've been without it being on startup, but this could cause trouble as Tony said.

 

I'd simply run a command manually, or set it up via arkservertools and do it that way... As stated, if the server is doing mad stuff with the savefiles, and you do something like this, it'll cause trouble.

Link to comment
Share on other sites

21 hours ago, Toni said:

Thats the reason why i don't use this wild dino wipe at restart function.  Changing the savefile while starting is really really risky.

For example if there would be a crash at startup (Like we had last weekend with Ragnarok), then its destroying your savefile because it is changing it at startup.

 

Here is the reason for the servers "downtime" / "no-response"

The problem is based on the save system of ark (Like i already reported 1000 times for other problems...And already posted a solution because Conan Exiles has a solution for it with the UE ! They key-word is SQLite Database ! ).

The freeze / non-response of the server is happening when the server is cleaning out the dinos out of the savestate (In RAM). The "database" system ARK uses, is completely trash. The server has too long to search / delete the entries in the savestate. 

Other problems like for example the freeze on Clientside which you get when you transfer a big amount of single items, is exactly based on the same thing.

 

For people which doesn't believe me (There are always such ^^ // Including wildcard), here is how to prove:

1. Run a wild dino wipe on a Ragnarok Server which has a 200+ MB save file --> Result: Server freezes for minimum a minute and kicks all players out

2. Run a wild dino wipe on an emtpy Ragnarok Map which only has the wild dinos and nothing other --> Wild dino wipe finished in a few seconds, no kick of players.

 

Other examples to prove the bad save system of ARK:

1. Run a server with a big savefile... but no players connected, only you for testing (So server fps is at max of 30 fps). 

2. Move many items from one container to another container (Have a look at how many items per seconds you can move)

3. Run a server with a very little savefile. Only place 1 storage box on the map and fill it. No players connected, only you for testing (Server fps at max of 30 fps)

4. Move many items from the container to another container  (Have a look at how many items per seconds you can move).

 

You will see that the empty savefile server is about 5 times faster than the big savefile server. Even if both has no players connected and the only difference is the savefile size.

The bigger the savefile... the bader the server performance by acessing something out of the save

 

More other examples:

It is reproducable with ALL ingame actions which needs to change something on the savesystem. For example remove big bases with command.

 

The SOLUTION for this problem (Listen Wildcard !! Please start to listen to me!!):

Rewrite the save system of ARK and use the SQLite possibilities of the UE !    (Or even better do it with MySQL ...so when can move the database away from our ARK servers).

The Conan Exiles devs (Using the Unreal Engine too) does use it. Result --> No laggs at server autosaves / No laggs at big item transfers  / No laggs when you remove many things from the map

It was simply wrong to program ARK (which has big save files) with this File-Based Save System  (The same for the Cluster directory).  Thats simply a way which things was programmed before 10 years.

Databases was invented because they are way faster (specially when it comes to large saves like ARK has)  than File based Save Systems like you use for ARK!

If you don't have the competence / capacities to rewrite your savesystem, simply hire one of the Conan Exiles Devs ! !

Yeah i know... this is months of work. But it would solve the most performance problems of ARK.   (Wouldn't it be nice to eliminate the autosave laggs/rubberbands all the servers out there (including your own) have???))

 

Simply I just wanted to .. don't know, maybe .. "like" this post but this site .. hmm .. So yeah .. I hope WC will do something about this rewrite thing! :)

Link to comment
Share on other sites

31 minutes ago, Thorium said:

Using SQL for storing game data in a real time game is a bad idea. It's terribly slow. You don't use a database for that at all. You just use common data structures like arrays, linked lists and hash tables. Especially hash tables can help a lot with performance.

Have a look at Conan Exiles (Using SQLite Database)... no save lagg, no inventory transfer lagg, Backups all 3 Minutes.

So before you post something... have a look like other developers done it with the Unreal Engine... instead of posting what your personal opinion is.

 

We all know --> Autosave = lagg 

We all know --> Transfering many single items = Freeze on client

We all know --> Wild dino wipes = server doesn't respond for a minute

We all know --> Transfering items into the ARK Data is very slow and you can't transfer several items at once

All this things are based on the file based save system they use. All this performance problems are related to the save system.

 

You ever seen a game server of other games which is doing a full save all 15 minutes intead of only writing the differences (incremental saves)? 

Ever seen an other game server which is saving all its data into 1 file?

 

Are you aware of? --> ARK dedicated is mostly single threaded  --> MySQL can use all cores and can be placed on a seperate hardware?

Are you aware of? --> With MySQL/SQLite you can do incremental saves which is 0.0001 % of the data to write than it actually is with the full saves?

Are you aware of? --> If your using a database, you can do some stuff asynchronously?  For example -> deleting the old dinos after a server wipe can be exectued later (Thats what actually is causing the non responseness on the server)

 

Link to comment
Share on other sites

3 hours ago, Toni said:

 

Are you aware of? --> ARK dedicated is mostly single threaded  --> MySQL can use all cores and can be placed on a seperate hardware?

Are you aware of? --> With MySQL/SQLite you can do incremental saves which is 0.0001 % of the data to write than it actually is with the full saves?

Are you aware of? --> If your using a database, you can do some stuff asynchronously?  For example -> deleting the old dinos after a server wipe can be exectued later (Thats what actually is causing the non responseness on the server)

 

 

MySQL is a terrible idea for a game, either WC would need to distribute it with the client for the single players, or expect the average player to be able to install and set it up, terrible idea.

However an embedded SQLite for the single players with an option to offload to MySQL for servers is not a terrible idea.  

ARK could run multiple threads for the database stuff and still single thread it's core code.  

Link to comment
Share on other sites

21 hours ago, Toni said:

Have a look at Conan Exiles (Using SQLite Database)... no save lagg, no inventory transfer lagg, Backups all 3 Minutes.

So before you post something... have a look like other developers done it with the Unreal Engine... instead of posting what your personal opinion is

It's less of an opinion. It's more a logical conclusion. Only because one developer uses it, it's not automatically good. However you do it, there is absolutely no reason for the inventory transfer freeze. As a developer myself i don't get what they did that causes this.

Using a SQL database for backend storage and accessing data that is not needed all the time like tribe logs is a good idea. Using it for everything is a very bad idea, you downgrade performance. SQL is a text based relational system, one of the slowest ways you can manage your data. It is very good for storing and managing huge amounts of data, if you don't need real time access.

Link to comment
Share on other sites

18 hours ago, Thorium said:

It's less of an opinion. It's more a logical conclusion. Only because one developer uses it, it's not automatically good. However you do it, there is absolutely no reason for the inventory transfer freeze. As a developer myself i don't get what they did that causes this.

Using a SQL database for backend storage and accessing data that is not needed all the time like tribe logs is a good idea. Using it for everything is a very bad idea, you downgrade performance. SQL is a text based relational system, one of the slowest ways you can manage your data. It is very good for storing and managing huge amounts of data, if you don't need real time access.

The best example to measure the actual binary flatfile savesystem is -->  Move some items into the cluster.

Is the performance nice there? No it isn't.  Would an SQL be way faster by adding/removing some items? --> For sure

An other example --> Wild dino wipes  -->  Now generates a timeout for about 60 seconds.    How long would a database have to DELETE * from wilddinos; ?  A few milliseconds....not 60 seconds!

 

Maybe your right in general. But the actual state of the game shows that the implementation of this binary saves is very inperformant.

Link to comment
Share on other sites

Guest BubbaCrawfish

This is all going off-topic at a hell of a rate... Do any of you know what method they even use? And if not, why is it even being brought up in this thread which is about network protocols...

I'm pretty sure that with Maels issue of 20 minute crashes, this doesn't even come to play as the problem they are facing, is likely to be a hardware issue of some sort.

Link to comment
Share on other sites

  • 3 weeks later...
  • 6 months later...
On 12/31/2015 at 6:47 PM, Drathek said:

We are still currently investigating performance and stability issues that are related to use of raw UDP socket server connection mode. We recommend server hosts, and admins disable this option by removing "bRawSockets" from the launch parameters until issues are resolved. This will result in your server using the Steam peer-to-peer API instead.

Original: http://steamcommunity.com/app/346110/discussions/10/517142892069785665/

Interesting. I switched to this option quite a long time ago. It seems to perform better and users report a very good ping without lags. Are there some more technical insights? I know the difference between UDP, TCP, P2P and via another Server.  If that's the only difference then UDP packets can get lost but it doesn't really seem to happen much.

Link to comment
Share on other sites

  • 2 months later...

Just chiming in again. I do not get these freezes on my cluster, and when we play around spawning in dinos with the "raindanger" command and do "destroywilddinos" repeatedly, my server sees no spike in CPU usage and nobody lags out. I have been using UDP since it was added, since the Unreal Engine is designed around it. I run five servers on an older PowerEdge 2900. It has two quad-core Xeons and 64Gb of ECC RAM, as well as an eight-disk RAID10 array.

Link to comment
Share on other sites

  • 6 months later...
  • 8 months later...
On 31/12/2015 at 17:47, Drathek said:

Todavía estamos investigando problemas de rendimiento y estabilidad relacionados con el uso del modo de conexión de servidor de socket UDP sin procesar. Recomendamos servidores host, y los administradores deshabilitan esta opción eliminando "bRawSockets" de los parámetros de inicio hasta que se resuelvan los problemas. Esto hará que su servidor utilice la API de igual a igual de Steam.

Original:  http://steamcommunity.com/app/346110/discussions/10/517142892069785665/

Is this just for who has an ark server?
Link to comment
Share on other sites

  • 2 months later...
  • 4 weeks later...
  • 3 months later...

Archived

This topic is now archived and is closed to further replies.


×
×
  • Create New...