FAF server on the move in January (downtime incoming)

TL;DR: We will have a large complete downtime at the beginning of January where we move to a new server. The exact date is not defined yet, as it depends on personal availability.

Many of you have encountered issues in the last months due to our server being overloaded when more than 1.000 users logged in. We spent dozens of hours trying to optimize and tweak the server and it’s services with little to no success.

During our investigations on server performance we saw that both CPU and disk IO was at it’s limit. This is a very unusual pattern as a server is usually either blocked by CPU or IO. But FAF follows no pattern as we have one machine running all kinds of services at once. Trying to relieve disk or CPU with larger memory caches did not help.

Our server was rented in late 2016 as an auctioned server at Hetzner. Even though it was fast enough, it was not the fastest machine available back then:

  • Intel Xeon E3-1270 v3 @ 3.50GHz [4 cores + Hyperthreading = 8 logical cores) which was released in 2011
  • 32GB of RAM
  • 2x 2 TB HDDs in a raid 1 (mirroring)

So as you can see CPU with 4 cores is somewhat slow, many gamers have much faster machines already. Also an HDD in a server is no longer state of the art especially for running databases (but it solved a particular issue which I will explain later).

While we were investigating something very positive happened over the year. The new AMD Ryzen 3 processor family has been launched and basically rolled up the desktop PC segment with fastest performance for a very low price. Even though the Ryzen 3s are actually not aiming at server segment, our current hosting provider Hetzner just recently launched a new product line based on these new CPUs at a very fair price.

So for just a few bucks more per month (45€ per month total) we would get a new machine with the following specs:

  • AMD Ryzen 5 3600 [6 cores + Hyperthread = 12 logical cores]
  • 64 GB of RAM
  • 2 512GB SSDs in a raid 1 (mirroring)

So that means an additional 50% of logical cores plus the technical improvements of the last 8 years! Without having exact comparisons it should basically double CPU performance. Moving from HDD to SSD will also have a huge boost on database performance. The additional RAM is a nice goodie, but not really needed for now.

So all ready, set & go for the server move? Unfortunately there is one drawback. As you have seen the disk size shrank from 2 TB to 0,5 TB. Looking at our servers today we will need slightly more than 0,5TB. The biggest part of that is our replay vault and part of that seems to be duplicated, so some additional investigations required there, maybe we can reduce that. In worst case we would drop parts of the oldest replays which are most probably broken anyway. Another part of the solution is adding a new compression to the replay format, which would reduce replay size by ~20% (more information here).

Some of you may ask why this has to be done in the holiday season – the favorite time of the year to play FAF. The answer is fairly simple: I can only do it when we have vacations ourselves.

A lot of preparations will be done beforehand, so that we can keep the downtime as small as possible. Some tasks cannot be prepared however. And that is exporting the database on the old server and importing it to the new server and also updating the DNS records to the new servers. In theory the latter is done in seconds, but they way DNS works it might take up to 24 hours until everybody will be redirected to the new servers.

I’m aiming for the transition to happen somewhere between 3rd and 5th of January. This date is just an estimation.

Feel free to discuss this topic in the forums here. (Unfortunately the forum won’t be available during the move.)

Download and connection issues (IP v6)

Dear commanders,

we are receiving a lot of complaints about unavailable services. In detail the What’s new? tab in the client, downloads from our content servers (maps, mods, replays, etc.) are broken for all users using IPv6.

We are aware of this issue and sorry for the inconvenience. While fixing the server setup might take a while, a confirmed workaround is deactivating IPv6 on your computer. Our fellow contributers have created a walkthrough how to do that on Windows 10 in our wiki here (use on your own risk).

Nomads Patch V90

Hello everyone,
Today its time for the first Nomads patch of the year – Nomads V 90


Since the last patch, there have been 87 commits, and we have quite a lot more things stacked up for next time. Its hard to guess the time scales now, but we might be adding a new unit every patch, possibly even more, so keep track!

But to keep things going, we need you! Yes, you! To help out with the mods development, as well as just keeping up to date or even complaining, head here:

Nomads discord server: https://discord.gg/eV3wmAX

You can play the newest version right now by selecting Nomads from the game type list inside the create game dialogue, enjoy!

Changes for this version:

New T1 tank model:

This has actually been a little bit of a surprise model. At some point we made the decision to make the T1 tank non-amphibious, and planned a model for it. But it was a very low priority and we didnt expect to get one any time soon. But then Redmelevolence just went and made one! After a bit of cleaning up, its now in the mod, which is his first major contribution to it. Congratulations to him for keeping up with the work until it was finished. Also: thanks to Armaster and Marlo for working on it during various stages.

This guy doesnt have any special abilities, but there is an antenna that rises up when it gets a target! So keep an eye out for that little detail. We hope you enjoy seeing the new acu helper as much as we do!

As usual, we are always looking for new developers to help us with our work. If you are interested in contributing, or even leaving a suggestion, or simply want to keep an eye on our progress as we continue our work on this mod, feel free to join our Discord server here: https://discord.gg/eV3wmAX

We look forward to hearing from you, and thank you for your continued support!

Additionally, if you are looking for help, advice or support on modding supcom in general, or how to do what we do, you are more than welcome to join and ask questions. We are also offering any content and source files we created, so you can get all the resources we use, some of which is extremely exclusive!

And very lastly, you can see a whole more info in the forum post, which also contains all the other changes we made, and much more: https://forums.faforever.com/viewtopic.php?f=15&t=14315&p=182242#p182242

Sincerely,
The Nomads Development Team

Nomads Dev Diaries #12

Hello everyone,

A few weeks back we about Nomads in 2020, and if you havent yet read it, read it now. Today we are talking about a couple of difficult models, and some of our advances in the code. Lastly, check out our development streams.

As always, this content you can spy ahead in the Nomads Development Discord, so if you want to get the exclusive info before everyone else, chat to the devs, ask questions or even contribute, you can do that here.

Static Anti Air

Previously, we have worked on the models for the anti air and point defence turrets. These would end up as secondary weapons on various units, as well as T1 static defenses. One longstanding issue was that there was no base to these, so thats what we have been working on.

T3 Amphibious Tank Model

Since last weeks sketch, we have got a nice little model from Marlo. Its pretty simple so far and has a long way to go, but it might be finally a tank concept that might get finished. We will see what the future holds, as always. Meanwhile you can have a look at the model and the sketch its based on.

Lots of Refactoring

Recently we have been working pretty hard on cleaning up the code for the mod. The existing code base has been rather old and had many fixes on top of fixes on top of fixes. This led to it being pretty bloated, laggy and hard to maintain. So throughout the last year or so we have been fixing it as we went along. In the past week or so thanks to speed quite a lot of stuff got fixed. Here is what was done so far:

  • A bunch of one line functions has been replaced, making the game run a little bit faster overall
  • The building beam effects of all nomads units were rewritten, making the code a lot smaller, and a whole lot faster. We previously fixed a similar bug which we fixed by reducing the effects of these beams, but now things should be even faster, so thats great.
  • Quite a lot of files got cleaned up, with random lines removed, random file imports, old useless code deleted. Again, things should become faster thanks to that.

Theres more coming in that department as usual, so great job speed! Be sure to thank him when you see him as well.

And thats it for now! If you want to see top secret stuff, join the discord server where you can chat to the devs, ask questions, spy on the process and get all the exclusive goodies and more! Also feel free to leave feedback either on the forum thread or in the discord. Remember, feedback is very important!

Lastly, we are always looking for more contributors, so if you are interested in creating things, whether thats code, 2d art, models, music or anything else, do get in touch! We also provide all the resources and guidance needed, so don’t worry if you think you are missing any skills, we will be glad to see you!

Tournament News

Contents:

  • Restriction Affliction 1v1 Tournament
  • Mapping Tournament #4 (20×20)
  • Third Season of FAF Ladder League (RESULTS)

Restriction Affliction 1v1 Tournament

Bothered by 1v1 games with tons of spam, or you feel like being hard on yourself? Sign up for this unique 1v1 tournament, where all is a standard affair, except you have a 125 max unit cap in total. There will be separate brackets for above and below 1700 (this may change in the future.) If you feel up to the task, take part of a much tighter, and challenging competition!

Tournament Date: February 15th at 16:00 UTC.

To Sign Up: https://forums.faforever.com/viewtopic.php?f=26&t=18741

Mapping Tournament #4 (20×20)

The next mapping tournament arrives with FtXCommando taking charge, but following biass’ structure and rules from his previous series of mapping tournies. This one will feature maps created on 20x20km sizes, and there will be a panel to provide judgement on the quality of the submissions.

It is advised you carefully read through the forum OP to understand the rules, to know when the deadline arrives, and to heed the advice provided on what kind of map you should be making.

Submission Deadline: April 1st at 12:00 UTC.

You must submit with the name of the map, and are welcome to include any other information that you find critical for its admission.

Forum OP: https://forums.faforever.com/viewtopic.php?f=26&t=18762

Third Season of FAF Ladder League (RESULTS)

Season 3 of the ladder league has finally concluded, leaving us with our ladder champions as well as our faction kings and princes! Congratulations to our winners:

The Champions of Each League:

  • 1800+: ThomasHiatt with a total score of 43!
  • 1300-1800: Shade- with a total score of 101!
  • 800-1300: dickdawkins with a total score of 196!
  • 300-800: coca with a total score of 124!
  • <300: CrazyChaos with a total score of 50!

The Kings of Each Faction:

  • Aeon: ThomasHiatt with a 97% Aeon pick rate and a score of 43.
  • Cybran: concuniorNoob with a 77% Cybran pick rate and a score of 20.
  • UEF: silentNoob with a 53% UEF pick rate and a score of 35.
  • Seraphim: No one lol.

The Princes of Each Faction:

  • Aeon: Achiraaz with a 100% Aeon pick rate and a score of 41.
  • Cybran: JeanLu with a 100% Cybran pick rate and a score of 92.
  • UEF: Swkoll with a 100% UEF pick rate and a score of 48.
  • Seraphim: CRISPYISARETARD with a 76% Seraphim pick rate and a score of 72.

We look forward to seeing you for next season! Next Season will begin in March

Forum OP: https://forums.faforever.com/viewtopic.php?f=26&t=18042&start=20#p181598

Nomads Dev Diaries #11

Hello everyone,

A few weeks back we talked about Nomads in 2020, and if you havent yet read it, read it now. Today we are talking about even more models, and showing you a couple of sketches. Additionally, Development Streams Are Available.

As always, this content you can spy ahead in the Nomads Development Discord, so if you want to get the exclusive info before everyone else, chat to the devs, ask questions or even contribute, you can do that here.

T3 Amphibious Tank

The T3 light tank for nomads is currently a hover tank. This isnt working out that well for it, and so we are planning on making it go along the sea floor instead. This means that it needs a new model for that to work, and here is what we have come up with so far.

This particular model has been quite a bit of trouble, having gone through several iterations already, and none of them being quite good enough. This last one seems to be promising though, so we might end up with it looking like this, or not. Time will tell. The idea for this model was to focus a little bit on what makes it amphibious and turn that into the characteristic of the unit. In other words, we are going to have a fantastic submerging animation. We expect everyone to be amazed when it emerges out of the water onto the shore and unpacks its turret to shoot at their enemies, so stay tuned.

Also, there is a recording of the making of these two scribbles, enjoy:

And here you can see some of the older designs that didnt make it. There are a lot more than just those, one day when the unit is finished we might post a complete list, but for now its a secret!

Many More Models

Continuing his crusade from last week, Marlo has been very busy drowning everyone in his work, so its only fair to show some of it off to you guys. Not everything here is going to get into the mod eventually, but some of it is, and they are all nice to look at, so enjoy.

And thats it for now! If you want to see top secret stuff, join the discord server where you can chat to the devs, ask questions, spy on the process and get all the exclusive goodies and more! Also feel free to leave feedback either on the forum thread or in the discord. Remember, feedback is very important!

Lastly, we are always looking for more contributors, so if you are interested in creating things, whether thats code, 2d art, models, music or anything else, do get in touch! We also provide all the resources and guidance needed, so don’t worry if you think you are missing any skills, we will be glad to see you!

FAF Newsletter: 7th of February, 2020

Welcome to the first of the official newsletter for FAF in 2020! We intend to have this article cover multiple topics relevant to the date posted: future, present, and/or past. We will post the newsletter biweekly, though this may change to be weekly if there are a lot of events going on.

Team Matchmaker

Team Matchmaker testing; This does not reflect the finalized UI set up, and is undergoing further changes.

If you have skimmed through the forums, you may have come across TMM, or Team Matchmaker, a major planned feature in the making that will add a whole new game mode on the client. TMM is a 2v2 matchmaking system, ideally setting either a single person or two people in a party, to play against another team, functioning as an automated custom matchmaker with its own map pools incorporated.

As for the reason for implementing this new feature, there has been two major points raised for the importance of TMM. The current custom game environment has created a sub-par experience for both the new and the old players in our community. The new player experience on FAF is defined by one word in this system: restriction. FAF has had an immense issue with player retention for years due to the reality of new players both finding team games preferable yet having no simple matchmaking tool. This has forced the new player experience on FAF to be at the mercy of the custom game system in which they are often kicked as they joined a lobby with a rating restriction, kicked because they joined a private game, and even kicked from all welcome games because of their lack of rating. On the other hand, experienced players suffer from global rating both encouraging the exploitation of maps and alienating the community into subsections that rarely engage one another. 

A Team Matchmaker will enable entry-level players to access novice team games easily and reliably, minimizing all current barriers from the start. Alongside this, introducing a new rating category tied to matchmaking will allow players to branch out from being rooted on single maps or positions and experience other aspects of FAF.

Regarding its current status, TMM is nearly complete on the server front; there is work to do on finding out ways to party with uneven teams, as well as figuring out how to set game spawns and conditions through the matchmaker. The larger issues remain with debugging, and more importantly applying a user-friendly interface. Once TMM reaches a state of maintaining a highly functional system, we intend to expand matchmaking up to 4v4, and make the experience as comfortable as possible with right adjustments to the UI.

When it becomes releasable, we will have a lot of work spent on fixing bugs. If you would like to help test for these, sign up for the discord tester role (“!subscribe Tester” in #faf-testing channel) on the official server. If you have experience utilizing java/python, you can contact Askaholic (server-side handling) and Geosearchef (UI Implementation). Otherwise, feel free to contact a moderator if you would like to get in touch with our Zulip development chat.