• Announcements

    • Jatheish

      Note for players regarding Ascension (please open terminal/tribute before ascending)   02/04/18

      With the latest server update on PC (v276.493), if you're going to attempt ascension, before doing so please make sure you've opened a supply crate/transmitter/obelisk/ basically anything terminal/tribute inventories. It's a temp workaround to characters being lost when ascending whilst we're investigating character issues further.


  • Content count

  • Joined

  • Last visited

  • Days Won

  • Feedback


TheRightHand last won the day on November 20 2017

TheRightHand had the most liked content!

Community Reputation

266 Tribe Leader

About TheRightHand

  • Rank
  1. Our dedicated servers cap out in the worst cases at 13-14 GB each. I don't know what could be causing your servers to be requiring that much per user, but even at 70 players maxed out our servers sit at about 13-14 GB of RAM.
  2. Hey everyone, it's me again, that guy who says stuff, and then tries to make sure other stuff gets done. It's a crazy time! So, here's what's going down for sure: Advanced Auto Turret - BobCorp has provided me with authorization to distribute the BobCorp Automated Laterally Attenuated Nano-Cell Electronic Defense system. This is a new turret we'll be introducing along with these changes that will essentially be a bigger, beefier turret, allowing for people who still need that damage density to populate their turret slots with turrets that are much more powerful. We're aiming for about a 4x increase in overall effectivness and cost for these things, along with disabling their use on rafts/boats. Construction costs and ammunition costs would be similar. We opted for a new turret because of a few things: - It conveys the power of the new turret clearly without needing to add any new functionality to anything else. - It doesn't create a sudden need to change the balance/functionality of existing turrets in any way, and thus rebalance all sorts of other stuff that could potentially break. We're aware we have already done a lot of that, let's not push it farther. - Existing turrets don't get a sudden, massive buff. - We can scale the cost/maintenance/upkeep of the new structure without inconveniencing everyone who isn't pushing these upper limits. - If we need to introduce any specific kinds of modifications to this new turret, we can do it without impacting anything else. So yeah, that's why that. In addition, this stuff: We're coalating feedback that we're getting along the way to develop a better sense of what exploits may exist or severely disproportionately effective tactics might need additional adjustment. For example: - Stego Armor Plates are going to have to block less damage from either bullets, or turrets specifically. - Veggie Cakes may need a more prohibitive delay on their use in combat. - Adding knockback to either the new turret, or both new and existing turret to offset specific movement speed + shield + rocket/c4 configurations may be in order. These are just examples of things that we'll be seeing how they play out and what kinds of impacts they might have outside of just the general pvp game (for instance, veggie cakes and Therizinos are a good way to fight the dragon right now.) and whatever we decide to act on, I'll make a post about it sometime before December 5th. I also want to take this opportunity to address a couple of things I've seen in comment threads: With the exception of some really really large dinos (dinos like the Bronto) we disable idle animations on dinos on the server. Idle dinos also only tick once every 4 seconds or so instead of every frame. Idle dinos, while expensive, are only expensive in a general sense. Things like their animations have next to no cost on the server, and we use paralellized animations, which means most animations aren't even run on the main game thread (They're much faster.) Dino head tracking is client-side, not server side. It has no cost on the server. The vast majority of the servers that people play on are very expensive, custom-built servers with top of the line 8 core/16 thread cpus running at 4ghz, 64 gb of ram. We pay out the nose to make sure your servers are powerful. The ARK Server also only uses 2 threads on the CPU. This is because the version of Unreal that we built the game on did not support multi-threading/paralellization. We have integrated Paralellized animation, and networking, so our networking overhead and cost of animation work are done on a 2nd thread. We're doing more work to try and thread more elements of the server, but it is incredibly complex and difficult work to do. Running 3 instances per server only takes up 6 cores at most. Having less instances per box would have no impact on the performance of each individual server. We do a lot to make the game run as smoothly as possible, while still enabling the kind of freeform, open-world experience that the game was designed to be. Anyways, I'll do a follow-up sometime later in the upcoming week about any things that we've decided we're going to do for sure. For now, TLDR: Adding a new, more powerful turret that'll be able to replace about 4-5 of our current turrets. Making Stego Armor Plates take more damage from turrets. Still collecting feedback and looking into additional measures. Thanks for your time, and your patience. - The Right Hand
  3. Turret Changes: A Technical Talk about why.

    Currently, the architecture of the version of Unreal we started building on is single-threaded. That means it will only use one CPU thread in order to do all of the stuff it does. A while back, we added paralellized networking and paralellized networking to our build, so that networking actions and animation rendering could both take place on multiple threads, increasing the number of total threads used by the server to 2. This was months of work. While we're investigating what else can be done for paralellization, the machines that about 70% of our servers run on are custom built baremetal machines with 8 core, 4+ghz processors, and we run 3 instances per machine, with 64gb of RAM per machine. This means that at most, all 3 servers are using 6 cores + 2 cores for our reporting, metrics, operating system overhead, etc. We pay an ENORMOUS amount of money for our servers. It's not that the servers are weak or cheap. It's that the game is just... expensive. It has to do a lot of stuff, all the time.
  4. Turret Changes: A Technical Talk about why.

    It's not well highlighted, but it's in the post. Even if we literally did nothing but let the server calculate turrets, the server fps would still be 2.5 (in a worst case scenario) The actual gameplay tick cost for EVERYTHING IN THE WHOLE GAME except turrets on a server that is well trafficked and has 70 active players on is about 240-250ms Turrets alone can be 400-500ms That's twice the cost EVERYTHING ELSE IN THE ENTIRE GAME There's no way to "make up the difference" by changing some other part of the game.
  5. Turret Changes: A Technical Talk about why.

    Attacks are done server-side, and in certain extremely large dinos they are networked, but the vast majority of them are clientside.
  6. Turret Changes: A Technical Talk about why.

    Dino idle animations and head-scanning are done clientside-- they cause no load on the server.
  7. Hey guys, sure has been a crazy day hasn't it? So, I know we have that megathread over there, and guess what my weekend gets to be (It's reading that and every other thread.) but I wanted to make a new post here just so that information was in a good, easy place to see. Let's just get right into it: This is some profiling output from a fun little tool we made, which analyzes the server and finds each thread that is running and times how long it takes for the server to tick. It does a lot more than that, but that's what you need to know to understand this: Total Duration 765.60 MS 1.31 FPS StructureTurretBaseBP_C : 7977 234.98 MS StructureTurretPlant_C : 9722 119.4 MS This is turrets. BaseBP is normal metal turrets, Plant is SpeciesX. Now, with this information, you need to have some other information: Since about October or so, we saw a MASSIVE spike in the size of stalls and poor performance on our dedicated servers, but only some of them, most of them Ragnarok Servers. So we investigated, and we found that an interesting change in the meta for base building had occured: People were making new, super-dense bases with insane turret counts, far beyond anything we'd ever seen before. The bases of these types were on all of our servers, but primarily belonged to Alpha Tribes who had set up bases on the Ragnarok servers. Similar setups caused equally bad performance on Center and Island servers in the rare cases we found them. I want you to understand what all of this means to us, so I'll try to be brief. Turrets, were very suddenly being used in a configuration that was atypical, and up to 7x the number of turrets we would see on any other server, even if it was a highly populated, well trafficked server. There were definitely other cases of densely packed bases, but this was higher by many magnitudes. Let me emphasize now, that for the last two years, turrets have been a pain in our butt, but we have not encountered people making monstrosities like this. Every alpha tribe in the game has existed just fine, and gotten along well enough without these insanely dense crazy bases. So, we looked at what we could do about it. Let me wrap this all up in context, these things I'm going to put here are facts: Our turrets are highly optimized for what they do with respect to gameplay. They use a very fast bitmasked octree overlap to return all of specific types of actors (excluding irrelivant actors) in the aoe that they defend, and then act according to the settings you configure, early-outing from any extra calculations or wasted instructions if their shot is invalid. We could make that area they search smaller, which would be faster, but then they would need correspondingly smaller ranges. We could make them search for less things in their overlap check, but then you'd have turrets that didn't fire on some things. We could make them acquire targets less frequently, but then they'd be much less accurate and in many cases would not pick up things like rocket projectiles that entered their range between scans. We could make them scan at different ranges for different things, but then their functionality would be weirdly ambiguous and what things would we reduce scan range for anyways? Anything that a turret should be shooting at is important. Additionally, even if we did one or all of these things, the per-frame cost of each turret would not be greatly reduced (fractions of milliseconds) and 15k+ turrets would still cripple servers. Another truth is that in the worst case, servers were ticking at as low as 1 frame per second. That means that no matter what you're seeing in game, the server can only update the whole game state one frame per one second. That is the worst our servers have ever run, and is completely unplayable, in every way. No matter who you are, I think you can agree that it's not enjoyable to play the game like that. As such, we are left with a dillema: We can reduce the number of turrets that people can place, and attempt to recover some of the performance loss that happens because of them, or we can not, or we can try and find some medium where we save as much performance as we can, while still allowing players to defend themselves. On a server with 70 people playing on it, after reducing turret density, this is the new framerate on the server: Total Duration 222.86 MS 4.5 FPS Now, this is a worst case scenario, but the difference is absolutely clear to us, and one thing that we cannot under any circumstances forget, is the impact that servers running at 1 fps has on all of the players on the server. 4-5 fps is actually playable-- it can be a little bit choppy, but it's not infuriatingly awful. Okay, so, if you're still with me, you understand what the raw numbers are that went into this, and I'll provide a to-now TLDR: A certain method of building bases with turrets caused server fps to plummet to historic lows. The culprit is undeniably turrets: in some cases costing between 400 and 500ms per frame on the server tick. Our turrets are already extremely well optimized, but players are placing tens of thousands of them. That's too many. Our changes showed a change of 1.1 -> 4.5 fps. Reducing their numbers was now not a question of "if" but of "how much". So what do we actually do? Well, we try to solve it. And we try to talk about how best to solve it. Are there going to be some edge cases we have to deal with? Sure. Will we maybe have to adjust the balance of some things? Probably. We're letting you know weeks in advance as part of a method of opening up that communication channel. So that you can make decisions about what to do well ahead of time. Crunch numbers. Adjust strategies. Yell at us, and let us know what some of the things that are important to you guys are that we might not be thinking of-- hundreds of thousands of people play this game in all sorts of different ways every single day, we simply can't know what all of those ways are. Now you know the technical issues we face. There is no other place to "save" 500ms of server tick time and let you just keep turrets how they are. There's no way to make turrets faster in any meaningful way, without fundamentally changing turrets and how they work. As an important note, this change will also make all of the turrets you DO have much more accurate and reliable, because the low server fps made them inaccurate and miss often. Kind of a compounding problem. https://gyazo.com/e307c4f0324ee5fc239862ace0365c38 This is a graph of the tick times of our worst servers, right off our backend. A delta time of 500 simply means that the server is locked at 2.5fps (it can actually run slower than that, but that's as low as our recording goes.) PC servers are far, far worse off than console servers, due to their increased caps on everything, and each and every one of these servers suffers enormously from turret-induced slowdown. PvE doesn't even come close. Out of the top 150 slowest servers we have, only 15 of them in fact are PvE servers. This is a pvp issue, and it's really bad. There is no truth to the claim that PvE runs worse. It is a myth. So let's talk about it. I am going to post this, and then I'm going to sit down and try to have dinner, and when I come back from dinner, I'm going to read more of what people have had to say. We want you to still be able to defend your bases and all of the stuff you've spent enormous amounts of time to make. We also want you to be able to play the game at higher than 1 fps. - The Right Hand
  8. Here's the things that did make it in: Memory Optimizations Texture Memory Reductions Mesh Optimization Animation Compression Optimization Here's the things that didn't make it in: GPU/CPU Optimizations Here's the why: Asynchronous compute and threading are stupid hard to implement into a non-threaded system and require tons of testing and iteration to get in, and every time we release highly experimental or potentially crash/stability effecting feature into the game for people, we get lambasted to no end about how we should test out stuff before we release it, and then we get lambasted to no end about how we should release our stuff. That's not a complaint, it's just letting you know that occasionally we decide to take the approach of carefully testing and iterating on a system internally before we release something that could have massive negative gameplay consequences in the form of crashes or other unknown instabilities, so that you guys don't have to deal with them, and that is one of the things that will have us hold on releasing something even if we thought we'd have it ready in time. This is literally a low-level re-write of a ton of core systems to use async compute. I know that doesn't translate well to a layman, but essentially we've taken a guy in a room who does most of the work of making the game run, and are trying to teach him how to split that work up among several different people now, all without anyone ever forgetting to tell any of the other people any of the important things that they need to know at every single moment in time or else the whole system explodes violently. For anyone who has worked on a team, or with teams of people, that's completely impossible. Fortunately computers are better at doing what you tell them, but you still can't make any mistakes dividing up all of that knowledge and letting other threads work on it and then making sure they're all sharing that info in every way they need to, without a lot of work. So yeah. It's hard. Sometimes stuff gets delayed, that's how game development goes. Just, normally you guys don't ever get to see any of that because it happens, and the game releases, and you never know about the 6 months that were spent trying to get a feature implemented when your technical team was so overloaded that they didn't have time for it, but then it gets in a few weeks before release. We're letting you guys see that happen. And it kinda sucks for everyone, us included. So enjoy the ride, every embarassing moment. Just try to be understanding that we're not doing anything scandalous, every company ever has had a feature they've had to wait months and months to get implemented, just none of them tell their players about it. Because they don't have any. Because their game isn't released yet. I love you. - The Right Hand
  9. It is an exploit, and the current method of doing it will almost certainly be patched in the future. We may or may not add in a legal/intended method of protecting the bottom of your boat, but the use of foundations for this is not intentional. - The Right Hand
  10. Rafts no longer viable in any way

    I'll be fixing the Leeds damage, it's accidentally scaling when it shouldn't :3c - The Right Hand
  11. Stuck in map. Can't die. Help

    To add to the above, you can jump or just punch the air and both of those actions will deplete your stamina. I know it's a bit of a bummer, but that's one of those hazards that comes with the island for now - The Right Hand
  12. Stuck Joining Session / PrimalGameData_BP

    We'll have a resolution for this issue coming as soon as we can push out a new patch. Sorry for the annoyance! We've been up all night resolving a number of these server browser issues