A bit of swearing, several facepalms, and 298 lines of code later, tags appear to have been restored to their former glory. This wins for most serious issue that was definitely caused by the server migration.
In other news, we've adjusted some settings to mitigate the 502 errors. Where you would have normally gotten a 502 error, the page will just take forever to load (60-80 seconds, max). We recognize that this is totally not an improvement. However, the important part is that every time this happens, the server will spit out a whole bunch of debug code in a special log for us to review. The "slow log" generates after 15 seconds of processing on our end, so you only have to wait 15-20 seconds if you want us to get an error report. After that, you're free to give up until we get around to fixing the particular database query affecting you.
I should probably explain why this is happening so suddenly. Previously, the database was moderately fast, but was not redundant, and couldn't scale. If something were to have happened to the database, there would have been no way to recover it. The database service was also running on the same server as the web service, which led to some resource contention.
We've given the database service its own, quite powerful server. The server is redundant in many ways; the most obvious being that backups are made at least daily, and the server is using redundant hardware. However, this upgrade required that we switch our database engine. For those interested in the technical details, we use MySQL, which has two major engines: MyISAM and InnoDB. We were previously using MyISAM, which is fast for certain things, but cannot scale to serve large websites, and cannot easily be made redundant. Backups are difficult. vBulletin, our forum software, was designed for MyISAM, and hasn't been well-tested on InnoDB. We've switched to InnoDB after making some changes so that we can continue to provide you with more consistent service. Although you might notice some errors, this is largely just InnoDB adjusting to the activity on our site and taking its time optimizing the database. Some of this is due to vBulletin not being designed for use with InnoDB. In the latter case, it's usually pretty easy for us to fix vBulletin once we know where the problems are. InnoDB can be significantly faster than MyISAM if used properly.
To help ease the transition, we have a third server running memcached. This is a caching engine: it stores cached data in very fast memory. It shaves several hundred milliseconds off response times to frequently visited pages, which is surprisingly noticeable. Go browse some old threads, and page load times will start to slow down. Unfortunately, we've had to decrease the extent to which memcached is utilized, as some of the software on our site wasn't using it properly, which was causing the truncated posts.
