TheRightHand

Administrator
  • Content count

    65
  • Joined

  • Last visited

  • Days Won

    3
  • Feedback

    N/A

TheRightHand last won the day on May 30

TheRightHand had the most liked content!

Community Reputation

255 Tribe Leader

About TheRightHand

  • Rank
    Naked
  1. Our dedicated servers cap out in the worst cases at 13-14 GB each. I don't know what could be causing your servers to be requiring that much per user, but even at 70 players maxed out our servers sit at about 13-14 GB of RAM.
  2. Hey everyone, it's me again, that guy who says stuff, and then tries to make sure other stuff gets done. It's a crazy time! So, here's what's going down for sure: Advanced Auto Turret - BobCorp has provided me with authorization to distribute the BobCorp Automated Laterally Attenuated Nano-Cell Electronic Defense system. This is a new turret we'll be introducing along with these changes that will essentially be a bigger, beefier turret, allowing for people who still need that damage density to populate their turret slots with turrets that are much more powerful. We're aiming for about a 4x increase in overall effectivness and cost for these things, along with disabling their use on rafts/boats. Construction costs and ammunition costs would be similar. We opted for a new turret because of a few things: - It conveys the power of the new turret clearly without needing to add any new functionality to anything else. - It doesn't create a sudden need to change the balance/functionality of existing turrets in any way, and thus rebalance all sorts of other stuff that could potentially break. We're aware we have already done a lot of that, let's not push it farther. - Existing turrets don't get a sudden, massive buff. - We can scale the cost/maintenance/upkeep of the new structure without inconveniencing everyone who isn't pushing these upper limits. - If we need to introduce any specific kinds of modifications to this new turret, we can do it without impacting anything else. So yeah, that's why that. In addition, this stuff: We're coalating feedback that we're getting along the way to develop a better sense of what exploits may exist or severely disproportionately effective tactics might need additional adjustment. For example: - Stego Armor Plates are going to have to block less damage from either bullets, or turrets specifically. - Veggie Cakes may need a more prohibitive delay on their use in combat. - Adding knockback to either the new turret, or both new and existing turret to offset specific movement speed + shield + rocket/c4 configurations may be in order. These are just examples of things that we'll be seeing how they play out and what kinds of impacts they might have outside of just the general pvp game (for instance, veggie cakes and Therizinos are a good way to fight the dragon right now.) and whatever we decide to act on, I'll make a post about it sometime before December 5th. I also want to take this opportunity to address a couple of things I've seen in comment threads: With the exception of some really really large dinos (dinos like the Bronto) we disable idle animations on dinos on the server. Idle dinos also only tick once every 4 seconds or so instead of every frame. Idle dinos, while expensive, are only expensive in a general sense. Things like their animations have next to no cost on the server, and we use paralellized animations, which means most animations aren't even run on the main game thread (They're much faster.) Dino head tracking is client-side, not server side. It has no cost on the server. The vast majority of the servers that people play on are very expensive, custom-built servers with top of the line 8 core/16 thread cpus running at 4ghz, 64 gb of ram. We pay out the nose to make sure your servers are powerful. The ARK Server also only uses 2 threads on the CPU. This is because the version of Unreal that we built the game on did not support multi-threading/paralellization. We have integrated Paralellized animation, and networking, so our networking overhead and cost of animation work are done on a 2nd thread. We're doing more work to try and thread more elements of the server, but it is incredibly complex and difficult work to do. Running 3 instances per server only takes up 6 cores at most. Having less instances per box would have no impact on the performance of each individual server. We do a lot to make the game run as smoothly as possible, while still enabling the kind of freeform, open-world experience that the game was designed to be. Anyways, I'll do a follow-up sometime later in the upcoming week about any things that we've decided we're going to do for sure. For now, TLDR: Adding a new, more powerful turret that'll be able to replace about 4-5 of our current turrets. Making Stego Armor Plates take more damage from turrets. Still collecting feedback and looking into additional measures. Thanks for your time, and your patience. - The Right Hand
  3. Turret Changes: A Technical Talk about why.

    Currently, the architecture of the version of Unreal we started building on is single-threaded. That means it will only use one CPU thread in order to do all of the stuff it does. A while back, we added paralellized networking and paralellized networking to our build, so that networking actions and animation rendering could both take place on multiple threads, increasing the number of total threads used by the server to 2. This was months of work. While we're investigating what else can be done for paralellization, the machines that about 70% of our servers run on are custom built baremetal machines with 8 core, 4+ghz processors, and we run 3 instances per machine, with 64gb of RAM per machine. This means that at most, all 3 servers are using 6 cores + 2 cores for our reporting, metrics, operating system overhead, etc. We pay an ENORMOUS amount of money for our servers. It's not that the servers are weak or cheap. It's that the game is just... expensive. It has to do a lot of stuff, all the time.
  4. Turret Changes: A Technical Talk about why.

    It's not well highlighted, but it's in the post. Even if we literally did nothing but let the server calculate turrets, the server fps would still be 2.5 (in a worst case scenario) The actual gameplay tick cost for EVERYTHING IN THE WHOLE GAME except turrets on a server that is well trafficked and has 70 active players on is about 240-250ms Turrets alone can be 400-500ms That's twice the cost EVERYTHING ELSE IN THE ENTIRE GAME There's no way to "make up the difference" by changing some other part of the game.
  5. Turret Changes: A Technical Talk about why.

    Attacks are done server-side, and in certain extremely large dinos they are networked, but the vast majority of them are clientside.
  6. Turret Changes: A Technical Talk about why.

    Dino idle animations and head-scanning are done clientside-- they cause no load on the server.
  7. Hey guys, sure has been a crazy day hasn't it? So, I know we have that megathread over there, and guess what my weekend gets to be (It's reading that and every other thread.) but I wanted to make a new post here just so that information was in a good, easy place to see. Let's just get right into it: This is some profiling output from a fun little tool we made, which analyzes the server and finds each thread that is running and times how long it takes for the server to tick. It does a lot more than that, but that's what you need to know to understand this: Total Duration 765.60 MS 1.31 FPS StructureTurretBaseBP_C : 7977 234.98 MS StructureTurretPlant_C : 9722 119.4 MS This is turrets. BaseBP is normal metal turrets, Plant is SpeciesX. Now, with this information, you need to have some other information: Since about October or so, we saw a MASSIVE spike in the size of stalls and poor performance on our dedicated servers, but only some of them, most of them Ragnarok Servers. So we investigated, and we found that an interesting change in the meta for base building had occured: People were making new, super-dense bases with insane turret counts, far beyond anything we'd ever seen before. The bases of these types were on all of our servers, but primarily belonged to Alpha Tribes who had set up bases on the Ragnarok servers. Similar setups caused equally bad performance on Center and Island servers in the rare cases we found them. I want you to understand what all of this means to us, so I'll try to be brief. Turrets, were very suddenly being used in a configuration that was atypical, and up to 7x the number of turrets we would see on any other server, even if it was a highly populated, well trafficked server. There were definitely other cases of densely packed bases, but this was higher by many magnitudes. Let me emphasize now, that for the last two years, turrets have been a pain in our butt, but we have not encountered people making monstrosities like this. Every alpha tribe in the game has existed just fine, and gotten along well enough without these insanely dense crazy bases. So, we looked at what we could do about it. Let me wrap this all up in context, these things I'm going to put here are facts: Our turrets are highly optimized for what they do with respect to gameplay. They use a very fast bitmasked octree overlap to return all of specific types of actors (excluding irrelivant actors) in the aoe that they defend, and then act according to the settings you configure, early-outing from any extra calculations or wasted instructions if their shot is invalid. We could make that area they search smaller, which would be faster, but then they would need correspondingly smaller ranges. We could make them search for less things in their overlap check, but then you'd have turrets that didn't fire on some things. We could make them acquire targets less frequently, but then they'd be much less accurate and in many cases would not pick up things like rocket projectiles that entered their range between scans. We could make them scan at different ranges for different things, but then their functionality would be weirdly ambiguous and what things would we reduce scan range for anyways? Anything that a turret should be shooting at is important. Additionally, even if we did one or all of these things, the per-frame cost of each turret would not be greatly reduced (fractions of milliseconds) and 15k+ turrets would still cripple servers. Another truth is that in the worst case, servers were ticking at as low as 1 frame per second. That means that no matter what you're seeing in game, the server can only update the whole game state one frame per one second. That is the worst our servers have ever run, and is completely unplayable, in every way. No matter who you are, I think you can agree that it's not enjoyable to play the game like that. As such, we are left with a dillema: We can reduce the number of turrets that people can place, and attempt to recover some of the performance loss that happens because of them, or we can not, or we can try and find some medium where we save as much performance as we can, while still allowing players to defend themselves. On a server with 70 people playing on it, after reducing turret density, this is the new framerate on the server: Total Duration 222.86 MS 4.5 FPS Now, this is a worst case scenario, but the difference is absolutely clear to us, and one thing that we cannot under any circumstances forget, is the impact that servers running at 1 fps has on all of the players on the server. 4-5 fps is actually playable-- it can be a little bit choppy, but it's not infuriatingly awful. Okay, so, if you're still with me, you understand what the raw numbers are that went into this, and I'll provide a to-now TLDR: A certain method of building bases with turrets caused server fps to plummet to historic lows. The culprit is undeniably turrets: in some cases costing between 400 and 500ms per frame on the server tick. Our turrets are already extremely well optimized, but players are placing tens of thousands of them. That's too many. Our changes showed a change of 1.1 -> 4.5 fps. Reducing their numbers was now not a question of "if" but of "how much". So what do we actually do? Well, we try to solve it. And we try to talk about how best to solve it. Are there going to be some edge cases we have to deal with? Sure. Will we maybe have to adjust the balance of some things? Probably. We're letting you know weeks in advance as part of a method of opening up that communication channel. So that you can make decisions about what to do well ahead of time. Crunch numbers. Adjust strategies. Yell at us, and let us know what some of the things that are important to you guys are that we might not be thinking of-- hundreds of thousands of people play this game in all sorts of different ways every single day, we simply can't know what all of those ways are. Now you know the technical issues we face. There is no other place to "save" 500ms of server tick time and let you just keep turrets how they are. There's no way to make turrets faster in any meaningful way, without fundamentally changing turrets and how they work. As an important note, this change will also make all of the turrets you DO have much more accurate and reliable, because the low server fps made them inaccurate and miss often. Kind of a compounding problem. https://gyazo.com/e307c4f0324ee5fc239862ace0365c38 This is a graph of the tick times of our worst servers, right off our backend. A delta time of 500 simply means that the server is locked at 2.5fps (it can actually run slower than that, but that's as low as our recording goes.) PC servers are far, far worse off than console servers, due to their increased caps on everything, and each and every one of these servers suffers enormously from turret-induced slowdown. PvE doesn't even come close. Out of the top 150 slowest servers we have, only 15 of them in fact are PvE servers. This is a pvp issue, and it's really bad. There is no truth to the claim that PvE runs worse. It is a myth. So let's talk about it. I am going to post this, and then I'm going to sit down and try to have dinner, and when I come back from dinner, I'm going to read more of what people have had to say. We want you to still be able to defend your bases and all of the stuff you've spent enormous amounts of time to make. We also want you to be able to play the game at higher than 1 fps. - The Right Hand
  8. Here's the things that did make it in: Memory Optimizations Texture Memory Reductions Mesh Optimization Animation Compression Optimization Here's the things that didn't make it in: GPU/CPU Optimizations Here's the why: Asynchronous compute and threading are stupid hard to implement into a non-threaded system and require tons of testing and iteration to get in, and every time we release highly experimental or potentially crash/stability effecting feature into the game for people, we get lambasted to no end about how we should test out stuff before we release it, and then we get lambasted to no end about how we should release our stuff. That's not a complaint, it's just letting you know that occasionally we decide to take the approach of carefully testing and iterating on a system internally before we release something that could have massive negative gameplay consequences in the form of crashes or other unknown instabilities, so that you guys don't have to deal with them, and that is one of the things that will have us hold on releasing something even if we thought we'd have it ready in time. This is literally a low-level re-write of a ton of core systems to use async compute. I know that doesn't translate well to a layman, but essentially we've taken a guy in a room who does most of the work of making the game run, and are trying to teach him how to split that work up among several different people now, all without anyone ever forgetting to tell any of the other people any of the important things that they need to know at every single moment in time or else the whole system explodes violently. For anyone who has worked on a team, or with teams of people, that's completely impossible. Fortunately computers are better at doing what you tell them, but you still can't make any mistakes dividing up all of that knowledge and letting other threads work on it and then making sure they're all sharing that info in every way they need to, without a lot of work. So yeah. It's hard. Sometimes stuff gets delayed, that's how game development goes. Just, normally you guys don't ever get to see any of that because it happens, and the game releases, and you never know about the 6 months that were spent trying to get a feature implemented when your technical team was so overloaded that they didn't have time for it, but then it gets in a few weeks before release. We're letting you guys see that happen. And it kinda sucks for everyone, us included. So enjoy the ride, every embarassing moment. Just try to be understanding that we're not doing anything scandalous, every company ever has had a feature they've had to wait months and months to get implemented, just none of them tell their players about it. Because they don't have any. Because their game isn't released yet. I love you. - The Right Hand
  9. It is an exploit, and the current method of doing it will almost certainly be patched in the future. We may or may not add in a legal/intended method of protecting the bottom of your boat, but the use of foundations for this is not intentional. - The Right Hand
  10. If you want to skip the why, and just get to the WHAT, go ahead and skip to the bold text down the page. Hey there friends! So, there's been a lot of controversy over this flier nerf, and we're glad to see that as always, our community is loud and engaging us in every way they know how-- including sending us image macros of a guy pooping on our names! Let's get down to brass tacks on this one so I don't waste your time: The big broad nerf hammer has been swung, and as the Senior Gameplay Designer on the project, it's my job to try and communicate as much and well as I can about what's going on with changes like these. I also have to make it a priority to iterate with you, the community, on stuff that impacts you, even if I'm not great at it all the time. That's part of our whole philosophy. I think the discussion of "why did you do it" needs a moment to talk about, because there is a lot of assumptions about why we did it, and the posts I have made elsewhere haven't traveled everywhere they probably should have. So, let's start there. We very much see fliers as needing to have distinct roles, and we feel this way about every dino, but each day only has so many hours in it, and there are a huge number of priorities that we have before we do a lot of balance stuff. In their state before this nerf bat, fliers were largely non-distinct, and speed was a critical issue across the board. The game had become largely "get a flier, rule the world", and this started as early as a Pteranadon. There were many PvP implications in flier speed, not only did it make snatching, C4 bombing, and Quetz Grenading huge, easy issues, but it had underlying repercussions in the overall pvp gameplay space with regards to trivializing the land dinos which we actually want to see in battle, and giving people the option to remain endlessly airborne with giant mobile chaingun/gatling/rocket platforms high above the world and out of range of any threats. There were also PvE implications. The ground game was essentially trivialized. You could live on the back of a Quetz perpetually after getting one, never even having to land... ever, really. 80% of the game was easily ignored because the air is safer, more straightforward, etc. Without taking forever to deep-dive into this stuff, the one thing that I think everyone can agree with, or that most people seem to agree with, is that "it had to happen", and "you guys should have just done it less". Both of these statements are true, but in the long run, the way we choose to make these changes are with broad strokes, followed by incremental buffs. That's how we work. There are other companies that do it differently, and that's good. For an in-development product, however, this is how we chose to do it. The end result will be the same, the only difference is where the outrage is felt. Either you guys get furious at us over several weeks as we nerf your fliers again and again, making them weaker and weaker, and you hate us for that, or we do it all at once and buff them, and you hate us all at once and then things get better. These are just two different design approaches. If this was a finished game, where right now we felt that creatures needed small incremental tweaks to get them to a place where we like them, we would probably have taken the Blizzard/Starcraft approach of little steps at a time. Now, let me take a moment out to address all of you on a very serious note: Don't ever let anyone tell you not to express yourself. If you have a feeling, if something we did made you feel a certain way, express it. You are not wrong to feel that way. It is your own feeling, and you have every right to express it. I've seen a LOT of people telling each other what is and is not okay to express, and I don't really like that. We have to have thick skin to be developers. We know sometimes you just need to say a string of expletives at us. That's fine. It's not CONSTRUCTIVE, but it's fine. Would we prefer constructive dialogue? Yes. We've used constructive feedback to shape the next balance pass on the fliers. But even just raw anger and frustration are fine to express, and we want you to. It lets us know what the general feeling and climate is, even if it doesn't help us move towards a solution for the problem. With that out of the way, the meat. Our overall goal with this re-balance is to define roles specific to each of the fliers. This is a monumental task for land dinos, and that is why we didn't do that now. There are like a hundred land dinos, and not many fliers, comparatively. The roles look something like this: Ptero - Fast, sprinting scout, with low hp/stamina, but quicker stamina recovery. Fastest flier in the game (except for wild wyverns). Argent - Tanky fighter, can scrap well and take some hits, high stamina/carry weight compared to all other fliers (except Quetz), while still maintaining some agility. Moth - "beginner" mount, lowest stats across the board for explicit travel or fighting, but a powerful special attack for specialized use. Pelagornis - Versatility flier with generally low stats, but unique functionality, and other special abilities to offset its lack of direct stat power. Quetz - Sky Bronto. Slow, extreme stamina, extreme weight capacity, good for prolonged sieges or for moving massive quantities of things to and fro. High health pool, but extremely poor maneuverability and very vulnerable. Tapejara - Tactical flier. Between the Argent and Ptero for overall stats, but with multi-mount support and latching abilities, it can carry well armed players in and out of the battlefield or to remote locations. Wyvern - Lord of the skies. Low stamina, moderate move speed, poor agility, but extreme power and versatility. This is how we see them conceptually, and the overall "feel" we want to aim for when we finally say "yeah, these are in a good spot". In the spirit of that: https://docs.google.com/spreadsheets/d/1hVE4Ub5ufePdkun4IYQf1oADPcvcfLx9IRpEbZVPLdc/edit#gid=160180008 This is a good look at the current values, the new values, and the deltas associated with the changes. Most of it is straightforward, but there are a few silly or convoluted variables in here: Rider Extra Max Speed Multiplier is the most important one, it effectively gives the associated dino a change in its movement speed when it has someone riding it. You can see we've increased these values across the board for almost all of the fliers, making them simply faster when ridden. So, all of the values shown here are essentially up for modification. This is a slightly trimmed version of the spreadsheet I use to calculate balance changes and overall impact and % deltas when making changes like these, I thought it would be cool to share it. The "New Values" will be in your patch tomorrow*. Afterwards, we will be watching for a few more days, and seeing how these changes play out in my own gameplay, in the live game, in overall metrics on the back-end, etc. Then we will do another pass with the rest of the team, and we'll continue to iterate until we feel everything is in a good place. Feedback us! If you have some thoughts on where these are going and want to take your own shot at proposing some values, go for it. I'm glad to look over anything that gets proposed, as long as you've got some reasoning behind it. This is not the final pass! We're going to be iterating on this internally and want your input as we go forwards. I'll try to do more of this stuff when we bring out large balance changes in the future, and do them beforehand, not after. It's a process, so it doesn't just end here. - The Right Hand *Wildcard Times are subject to change or horrible inaccuracies.
  11. Rafts no longer viable in any way

    I'll be fixing the Leeds damage, it's accidentally scaling when it shouldn't :3c - The Right Hand
  12. [10:24:28 PM] Jat You should work like a butler. [10:24:29 PM] Jat Alright [10:24:37 PM] Casanova Alfred makes all the money though [10:24:48 PM] Casanova and butlers don't do anything [10:24:50 PM] Casanova tbh [10:24:55 PM] TheRightHand In some universes Alfred owns WE [10:25:02 PM] Casanova exactly! [10:25:03 PM] Ciabatta nah [10:25:04 PM] Jat World elite? [10:25:10 PM] Jat The e sports [10:25:12 PM] Jat Organization? [10:25:14 PM] Casanova Wayne Enterprises [10:25:15 PM] Ciabatta he's got joint ownership with lucius fox [10:25:16 PM] Jat oh.. [10:25:18 PM] Jat idk what that is [10:25:21 PM] Casanova lmao [10:25:23 PM] Casanova usual [10:25:27 PM] Jat is tha a fresh prince [10:25:28 PM] Jat of belair thing [10:25:31 PM] TheRightHand Yes. [10:25:32 PM] Casanova yeah [10:25:33 PM] TheRightHand You did it. [10:25:33 PM] Casanova just like that [10:25:35 PM] TheRightHand Well done Jat. [10:25:44 PM] Jat isnt his name jeeves [10:25:46 PM] Ciabatta it's a jatman thing [10:25:46 PM] Jat the butler [10:25:47 PM] Jen He's evolving [10:25:49 PM] Ciabatta batman** [10:25:55 PM] Jat ohh yeah [10:25:58 PM] Jat i forgot batman has a butler [10:26:02 PM] Jat hes like a super butler tho isnt he [10:26:05 PM] TheRightHand No. [10:26:05 PM] Jat built a underground lab [10:26:06 PM] Jat and shit [10:26:08 PM] TheRightHand He is a normal butler. [10:26:15 PM] TheRightHand Batman made the Batcave. [10:26:15 PM] Ciabatta every comic book butler is a super butler [10:26:18 PM] Jat then who built the lab [10:26:19 PM] Ciabatta compared to rl butlers [10:26:23 PM] Jat then why the hell does he need a butler [10:26:25 PM] Jat if hes gonna build that lab [10:26:27 PM] TheRightHand To buttle. [10:26:28 PM] Jat why cant he build a robot [10:26:29 PM] Jat to do that shit [10:26:30 PM] TheRightHand Duh. [10:26:37 PM] Ciabatta because [10:26:38 PM] Casanova Lucius built all the tech [10:26:42 PM] Jat batman needs some help alright [10:26:45 PM] Ciabatta the robot doesnt know how to make good tea and scones [10:26:49 PM] Jat lucius robin's real name? [10:26:53 PM] Casanova LMFAO [10:26:54 PM] Ciabatta no [10:26:55 PM] TheRightHand lmfao [10:26:56 PM] Casanova I'm dieing [10:26:59 PM] Jat then idk who he is [01:39:00 AM] Casanova is anyone actually excited about Suicide Squad (jat don't even ask) [01:39:12 AM] Jat oh i know will smith [01:39:12 AM] Jat Is he playing himself? [01:39:12 AM] Jat why [01:39:12 AM] Jat what does will smith add to it? So yeah.
  13. Giga needs balancing

    There's still probably a pretty good chance you're crazy. Just, not about this. - The Right Hand
  14. Broodmother impossible?

    Yes exactly. We intend for all bosses to be unbeatable, and certainly didn't give the Broodmother super cranked stats because currently you can just lead her off a cliff into the water or tank her for a hundred years with beefy dinos. - The Right Hand
  15. Broodmother impossible?

    Just wait until we introduce the boss arenas and stuff, hoooo boy. But if you kill her you get a nice flag! - The Right Hand