Respawn discusses Apex Legends server issues in mammoth blog post
The truth hertz.
Server issues have always been a bit of a hot topic within the Apex Legends community, with players highlighting problems with lag, code:net errors and - more recently - DDoS attacks on ranked servers. To address all these concerns, Respawn has now published a developer deep dive aiming to "demystify" some of the problems, and explain what the team is doing to address them. You might want to put the kettle on, as this one's a long read.
Written by lead engineer Samy "Ricklesauceur" Duc, the blog post notes that Respawn has made some changes to how it tracks server issues. The introduction of a performance display in Season 6 meant players could report their problems more accurately, and Respawn also began tracking key performance metrics for players and servers. In recent seasons, Respawn also used a data science team to systematically analyse performance data. But despite all this, Duc said this approach means Respawn can still be slow to fix problems.
"From the moment you start noticing problems, it may take us up to two weeks to find a fix, and even more to deploy the solution if it requires a server patch," Duc said. "We can do better. We will do better. So let's talk about solutions."
In addition to a weekly report, Respawn is now moving to real-time alerting to allow the team to immediately fix hardware problems, investigate issues and work on patches. A new server-unique ID is also being added to the in-game performance bar, allowing Respawn to track down wonky servers a little faster. These changes are due to arrive sometime in the new season.
Duc anticipates that problems with slow-motion servers should be solved faster with this approach, as real-time detection will allow Respawn to take misbehaving datacentre machines offline faster. A new tool that can create RPROF files, meanwhile, should help Respawn optimise servers to reduce latency issues - as these files give the developers insight into what the server is doing during every frame.
Some solutions for more complicated problems are still in the works: Respawn hopes automation will help detect congestion points for packet loss issues, for instance, but it sounds like further solutions are still in the "early stages". Duc explained that shots that don't register, known as a no-reg, can be due to a bug in the game's physics simulation. These bugs can be difficult to detect, so Respawn is working on further automated tools to find these - and asks that players send in clips of detection issues to help track down these bugs.
Some problems like lag compensation, however, will never have a perfect solution. (This is often the cause for getting shot behind a door or wall, by the way.) Apex is designed so that players with low ping won't have an advantage over high ping players - a trade-off Respawn deliberately designed into the system. Duc noted this was "really important for rural players, or for players in regions where connectivity is unstable."
The infamous "code:net" login error is apparently a generic message that could mean any number of problems, and Duc said they often "have more to do with Respawn's services supporting the game". Steps have apparently been taken to reduce the likelihood of code:net bugs occurring, but Duc recommends reporting these messages to the EA help site.
As for the debate over tickrate (the frequency at which a server updates what the player is seeing), there is an extremely in-depth paragraph explaining why simply increasing the server tickrate for Apex Legends, like other online shooters have, wouldn't make a significant impact on performance.
"Put all this math together, and you realize that 20Hz servers result in about five frames of delay, and 60Hz servers result in three frames of delay," Duc said. "So for triple the bandwidth and CPU costs, you can save two frames worth of latency in the best-case scenario. The upside is there, but it isn't massive, and it wouldn't do anything for issues that are tied to plain old lag (like getting shot while in cover), ISP-level issues, or bugs (like with hit reg and slow-mo servers)."
The post adds that Respawn is investing in its online infrastructure and working with partners "from the server level to the ISP level" to improve the overall experience. A couple of issues that weren't tackled in the post, such as DDoS attacks, were addressed in a Q&A on the Apex Legends subreddit. Director of communications Ryan K. Rigney explained Respawn was improving tools to identify DDoS attacks, closing a number of loopholes, working with experienced partners to develop solutions, and investigating more serious consequences for those carrying out the attacks - including legal action.
Phew, that's a lot of info - and that's just the condensed version. If you're looking for the full mathematical and technical extravaganza, you can find the blog post over here.