Jump to content

Ongoing Server Instability & Crashes


balistikas
 Share

Message added by invincibleqc

Please report official server outages using this form: https://ark.gg/outage

If you lost in-game progress such as your character, etc. due to a rollback please submit a Support Ticket to get in-game assistance.

Thanks!

Recommended Posts

55 minutes ago, LUISF said:

It is ddos when a player came to ur server and admitte hes doing it and laughing about it and ask for a payment u have a point on that is also true but u know when is a ddos from a pc

yeah dude.. i can come to your server and tell you that. doesnt make it true does it? You are being suckered. When they come to your server you tell them to F off and knock the server down and then laugh when it doesnt happen. The fact remains that dozens of servers have been going down all weekend and it wasnt DDoS attacks. Why should yours be the exception?

Link to comment
Share on other sites

Our server thats PVE was PVP yesterday for almost all day. No one could stay online for more then 10 mins. No explanation for it. Then they took servers down for the patch. Our server, PVE The Island 48 NA was back up for 2 hours before crashing. Now its been offline since 3pm today. They are doing a fantastic job standing behind their product.

Link to comment
Share on other sites

3 minutes ago, Sausagedog85 said:

717 was down for 9 hours total, read the post first.. I said '8 hours later, and 717 is still down, 12 for others (meaning other servers)
I did make an update post stating when it came back online, but someone deleted it because apparently the words I used weren't allowed, (even though this is a community based on a 16+ rated game)

 

This server has had 5 MAJOR dowtimes (1hr+) in the last week and a half.. and about 60 (maybe more) crash/reboots/15m downtimes.

 

If you are relying on battlemetrics for stats, their system is flawed and generally tends to under report by upto 2 or more hours because, while the server is 'hanging' and people have been disconnected for an hour already, battlemetrics can still query the server IP therefore reports it as 'online' until it crashes completely..

 

No the server was NOT taken offline for maintenance, they give alerts for that, no alert was received, and this was not the first time this server has crashed under the same circumstances.

i never said they were taken down for maintenance. You should take your own advice on reading a post. If a server goes down and has a problem going back up, they have to fix that. that may take more than the normal 30 minutes. So if you were down that long, you had a problem and it wasnt something so simple as rebooting the server. You however seem to think that they just arent doing anything about it because they dont stop to inform you every time they take a crap. 

I'm very well aware of how battlemetrics works and i account for the delay in query times. 

Link to comment
Share on other sites

Dude, I am a Sysadmin and a network tech, and I also run and host my own game servers for a gaming community I've operated for the last 5 years.
 

Server repairs and updates require minutes of downtime, not hours, and you just don't test fixes on the fly on the primary server.
For updates and minor repairs you have a cloned server that you work on and test with, you make the repairs/updates on that server then pull the primary down for a file swap and reboot, this should require about 3 to 30 minutes downtime, not 9 hours,

In this instance it was not the physical machine that was down either, I was still able to ping the machine's IP from home.
In this particular case it meant that someone just had to force close and relaunch.
 

Even if it was anything more sinister the downtime could have been minimised by instantly replacing a damaged gamemode with a stable gamemode backup, and hooking in the old database, this would take minutes, not hours.

Someone is dropping the ball, that is why this happens, with a community this large there should be a sysadmin on duty (or at least on call) 24 hours a day to perform basic upkeep functions.

The issue with the server is not a code issue or a bug, the issue with the servers is hang, and eventual crash generally due to the server having insufficient memory to handle the task at hand,
(duping, spam, and overpopulation of spawnable items is a known cause of this, we solved this issue back in 2015 by just adding a 0.2 second wait delay to inventory transfers, disabling hotkeys, and implementing DDOS protection to prevent lag duping, then we set a realistic item limit and building plot size restrictions on structures and vehicles.)

Edited by Sausagedog85
Link to comment
Share on other sites

2 minutes ago, Sausagedog85 said:

Dude, I am a Sysadmin and a network tech, and I also run and host my own game servers for a gaming community I've operated for the last 5 years.
 

Server repairs and updates require minutes of downtime, not hours, and you just don't test fixes on the fly on the primary server.
For updates and minor repairs you have a cloned server that you work on and test with, you make the repairs/updates on that server then pull the primary down for a file swap and reboot, this should require about 3 to 30 minutes downtime, not 9 hours,

In this instance it was not the physical machine that was down either, I was still able to ping the machine's IP from home.
In this particular case it meant that someone just had to force close and relaunch.
 

Even if it was anything more sinister the downtime could have been minimised by instantly replacing a damaged gamemode with a stable gamemode backup, and hooking in the old database, this would take minutes, not hours.


The issue with the server is not a code issue or a bug, the issue with the servers is hang, and eventual crash generally due to the server having insufficient memory to handle the task at hand, (duping and overpopulation of spawnable items is a known cause of this, we solved this issue back in 2015 by just adding a 0.2 second wait delay to inventory transfers and disabling hotkeys, and we set a realistic item limit on structures and vehicles)

You seem to think your server is the only one out there. You understand that more than half the official network was crashing every other hour all weekend? Some servers werent even making it a full hour before crashing again. Some servers had to be put on new hardware. How much time should all that take when 10-20 servers were dropping at a time?

Link to comment
Share on other sites

1 minute ago, UncivilizedBehaviour said:

You seem to think your server is the only one out there. You understand that more than half the official network was crashing every other hour all weekend? Some servers werent even making it a full hour before crashing again. Some servers had to be put on new hardware. How much time should all that take when 10-20 servers were dropping at a time?

If you all your servers are crashing at once, you are doing something very, very wrong.
All of these servers should have been redeployed at reduced player access numbers to lighten the load until sufficient hardware updates where made.

But at the end of the day all of this boils down to sheer incompetence, cost cutting, or straight up ignorance.
In order for this to get this bad and go unchecked for so long, we must be ticking all these boxes
 
Unstable, untested update releases.
Insufficient hardware and/or network conditions 
Inactive live server administration (to control duplicators, flooders or modders.)
Inactive support, community and PR staff (to keep the community informed and updated.)
Non-existent staffing backup (for when regular staff are indisposed or unavailable)
Communication roadblocks (forum staff don't have access to sysadmin)

 

 

 

 

Link to comment
Share on other sites

I've heard reports in my discord channels of other servers showing up as PvP, not showing up at all, and some folks getting requests for a password for an official server. I'm wondering if there's some trolls messing around with the servers causing these and the DC problems. My rag server has been going down on a regular basis since the beginning of April, often for anywhere from 3-16 hours at a time. Really sucks!

 

Link to comment
Share on other sites

5 minutes ago, Sausagedog85 said:

If you all your servers are crashing at once, you are doing something very, very wrong.
All of these servers should have been redeployed at reduced player access numbers to lighten the load until sufficient hardware updates where made.

But at the end of the day all of this boils down to sheer incompetence, cost cutting, or straight up ignorance.
In order for this to get this bad and go unchecked for so long, we must be ticking all these boxes
 
Unstable, untested update releases.
Insufficient hardware and/or network conditions 
Inactive live server administration (to control duplicators, flooders or modders.)
Inactive support, community and PR staff (to keep the community informed and updated.)
Non-existent staffing backup (for when regular staff are indisposed or unavailable)
Communication roadblocks (forum staff don't have access to sysadmin)

 

 

 

 


I totally agree with your first two points. The anti-dupe patch was a disaster and we all know the servers are on poop-tier hardware. 
Live server staff have been hustling to replace crash/rollback losses and honestly, i have never seen them respond faster to it. People were getting the notice to schedule their appointment in under 24 hours. So i have to give them this one. 
As for the community and PR staff, Cedric posted yesterday on twitter that his father died. Maybe thats smoke, maybe it isnt but a lot of people have died from Covid. 
I cant attest to the level of backup/on-call staff  but you are probably right and yes, it does seem like forum staff are ill informed. 

Link to comment
Share on other sites

Dev Help

Can a Dev or GM or whoever does it PLEASE get me my base back, Was on Ragnarok71 and waiting to transfer when the server crashed. Now my character is gone. I have a main base with babies out and would like to get back on. My base is thecenter352 tribe name Tribe of Ranger, Player name Ranger cords are 46 86. Any help would be greatly appreciated. I didput a ticket in that was 2 days ago still no word

Edited by Ranger4
Link to comment
Share on other sites

22 minutes ago, Ranger4 said:

Dev Help

Can a Dev or GM or whoever does it PLEASE get me my base back, Was on Ragnarok71 and waiting to transfer when the server crashed. Now my character is gone. I have a main base with babies out and would like to get back on. My base is thecenter352 tribe name Tribe of Ranger, Player name Ranger cords are 46 86. Any help would be greatly appreciated. I didput a ticket in that was 2 days ago still no word

You wanna file a support ticket for that. 

Link to comment
Share on other sites

@Cedric @lilpanda

Genesis-official-Small-Tribes-93 Server has been getting DDOS for days. There are 3 tribes teaming, using meshing exploits, and DDOS the server any time you attack them or they attack somebody. We cannot connect at all for the last 4+ Days. 255 ping very often. Please, if you can do something about this DDOS or whatever is going on, it would be greatly appreciated. We have been unable to connect or play for more than 20 minutes. 6-Player tribe all unable to connect. Or 255 within 20 minutes if we do. This is a game that requires constant 24 hour defense and tasks such as rasiing babies that take many consecutive days in a row. How can servers without DDOS protection be the standard proceedure for this game?! I Love ark and I want to keep playing, but this is unreal, I cant even log or move if I do. This has been happening for YEARS on official and smalltribes servers that I play on. 

255Ping.jpg

 

 

 

20200420001742_1.jpg

20200420022907_1.jpg

20200420023053_1.jpg

20200420023554_1.jpg

Edited by Fronk
Adding screenshots
  • Like 1
Link to comment
Share on other sites

I honestly don't think it's DDOS,  it's a wide net of  official servers being affected and I'm pretty sure that would take a pretty monumental effort by many folks.  This feels more like a haywire patch with subroutines that are running and causing un-anticipated problems to me. When I first started coding in C I had a tendency to screw things up and many of my early programs suffered huge memory leaks and things that I mishandled.  This feels exactly like that.

 

Link to comment
Share on other sites

2 minutes ago, Rio4201 said:

I honestly don't think it's DDOS,  it's a wide net of  official servers being affected and I'm pretty sure that would take a pretty monumental effort by many folks.  This feels more like a haywire patch with subroutines that are running and causing un-anticipated problems to me. When I first started coding in C I had a tendency to screw things up and many of my early programs suffered huge memory leaks and things that I mishandled.  This feels exactly like that.

 

I agree. i think this was all side effects of the anti-dupe efforts. Possibly from the wide spread duping that is gonna happen during a heavy breeding event. I do not think those who have been screaming about DDoS attacks actually pay attention to the network as a whole or the would have realized that so many servers were going down

Link to comment
Share on other sites

  • bootstraptv changed the title to 899 gen two down 13+ hours
  • Joebl0w13 locked and unlocked this topic
  • Joebl0w13 locked and unlocked this topic
  • Joebl0w13 locked this topic
  • Joebl0w13 locked and unlocked this topic
  • Joebl0w13 locked and unlocked this topic
  • Joebl0w13 locked and unlocked this topic
  • Joebl0w13 locked and unlocked this topic
  • Joebl0w13 locked and unlocked this topic
Guest
This topic is now closed to further replies.
 Share

×
×
  • Create New...