DataBase Errors

[Eric Lyon]

Please bear with us, we have been getting hit hard with (possible) ddos attacks the last couple days on & off and still trying to block all the sources. Every time we seem to get it under control they switch to different ip ranges and start it again. It's very annoying I agree and we are working behind the scenes to try and eliminate this.

The first set of attacks appeared to be stemming from some of the chinese software bot spammers that were banned in the last 3 / 4 days. It was as if their software turned on at 10pm & ran till 1am pelting the servers trying to regain access automatically over & over sending hundreds of queries per second each.

Later, the incidents started happening at different times on & off with a few stable patches in between. So now we aren't sure if its chinese bot software's, a targeted malicious attack, or maybe something else.

Hopefully we'll be able to get all this resolved soon.

Sorry for the inconvenience and thanks for understanding.

Eric Lyon

 

[snowbird]

Means that these days database error may occur?Do not be surprised to encounter such a situation, is it?

It is not out of the ordinary for this to happen considering a forum like this that is database driven is using dynamic content that requires a database connection. Since dynamic content is more load intensive than loading static content, there are limitations placed on how many database connections there can be allowed at one time.

Ironically Eric, I saw you mention Chinese IPs and I recall about a couple weeks ago one of my servers being hit with failed brute force attempts on my ssh port from IPs in China. Apparently that is becoming a pretty popular location for obtaining a proxy.

As for DDOS, unfortunately it is an attack that is hard to prevent these days. There is no sure thing protection against it and the best way to mitigate them is to have lots and lots of bandwidth access to the backbone. DDOS is not really the problem here, the problem is all the botnets that are out there generating these attacks and with all the broadband internet connections made available to so many out there, it is becoming easier and easier to perform these kinds of attacks. Another method to help mitigate these problems is using Anycast which takes the server IP for namepros and routes it to several different servers hosted at several different hosts. In this method, people who hit the server's IP will be routed to the best available server based upon resource usage and location. This would help distribute the traffic through several networks vs. just 1.

Those who say to create a script to just ban the offending IPs, that can work for smaller DDOS attacks, but for serious attacks, the network is still being hit regardless. But that might be a good course of action at this time however since we are in fact seeing a database error which is a good sign. We are reaching the web server. There just appears to be too many database connections probably which is preventing some people from being able to access the dynamic content.

 

[Eric Lyon]

Are they still DDOSing NP? I am still getting the database errors.

Yes, We have narrowed it down to Bot attacks and are still working on a solution.

Its becoming quite aggrivating, especially while in the middle replying or posting, and hitting preview or submit to get one of those errors.

I agree, it's very annoying. Hopefully we can get this resolved sooner than later.

Sorry for the inconvenience,

Eric Lyon

 

[Ray]

Well they must be after your database so you could ask users to change their passwords in case they have the same ones on their registrar accounts. And also secure the database if it is not already.

The type of attack you are referring to Is a SQL injection where a hacker exploits an unsecure database in order to harvest information. This attack could also be used to bring down a particular web site by destroying data and tables, but since members have access at times and others don't and the database is completely intact, I do not believe it is that method.

What namepros is experiencing is a DoS attack, or Denial of Service. Which generally happens is a hacker uses a trojan to generate a gruesome amount of request from average household computers without the owners knowing. Generally, it is successful as the server is unable to process the large packet size and rate and will eventually time out or hang.

DoS attacks have been used against many reputable Fortune 500 companies to include google. There is no one way to prevent it with completely denoting access to the average joe.

In this case, which is generally common with forums, a spam bot is submitting hundreds of requests to create accounts and posts over and over and is simply not able to make it past the human validation.

To mitigate this issue, I recommend instating a load balancer with a dedicated SQL server. Increase the number of maximum connections in correlation with the processor, and use HTACCESS and iptables to block the current known block of ips. Install and update network tools such as APF and DoS deflate as previously mentioned. DoS Deflate will automatically detect DoS attacks and deflect them at the system level by adding them into the Iptables and not allowing the processor to accept requests. Also make sure the Linux kernel is up to date . A hardware firewall or NAT would be extemely helpful aswell.

I also suggest blocking namepros from the particular country of origin for a short time as the request will return back empty and eventually stop. I understand that this may create issues but it is the for the betterment of the community and would only be a short time. A member could access NP from an established VPN or proxy during this time.

Use this command in the kernel to find ip ranges with abnormal connections

#netstat -anp|grep tcp|awk ‘{print $5}’| cut -d : -f1|sort|uniq -c|sort -n

To address the issue that was brought up regarding passwords, I'm pretty sure that vbulletin uses bcrypt (blowfish encrypt) with a salt and a time deflect. For those that don't know bcrypt it is one of the strongest commercial grade encryption methods as it prevents the use of rainbow tables by using a salt and is time generated (key is generated at a certain rate in the salt which prevents Bruteforce) It's way more secure than md5, mcrypt, and the standard MySQL encrypting function.

That being said, unless namepros stores the passwords in plain text which I highly doubt under the worst circumstances our passwords are secure, even if using an older method.

Still one should always take precaution, regularly change passwords, and never link multiple accounts to one email address

Ray

Source: 5+ YR server and website administration.

 

[DN BROKER]

Thanks for the update Eric. I was wondering what the issue was for the last 12+ hours. I've issued an alert to the community on my blog and asked anyone with information to email me.

Cheers

 

[Nathan]

We have plans later to put everything behind a new firewall which should rectify the situation completely. For now I'll try to keep manually heading them off at the pass to keep the server loads down enough that we have fewer outages (fingers crossed), well, unless I'm sleeping, then it might build back up again till I wake up (gets some fresh coffee ready).

I'll look into the router option a bit to see what's available.

Thanks,

Eric Lyon

Eric,

Another way would be to make a script to automatically ban an ip after x amount of queries per seconds. For example; the server would block the ips that hit 100 queries (depending on the excessive amount of queries) per second.

 

[Bannen]

Good luck Eric

Wish there was a way to just shotgun all the ddos bots with one blast

 

[Eric Lyon]

well said snowbird and thanks for the ideas. That basically summed it up. early today we opened up 200 more ports just to see what the bots would do & sure enough, regardless how many ports we opened, they filled them up within 60 seconds.

I've re-enabled a few features that were collecting dust over the years that query a central spam database multiple vb users contribute to and initiated the auto blocking sequence again for known spam sources. It seems to be trimming down lots of the manual workload and leaving me with a more manageable amount of bots that slip through the cracks (never reported before).

The downside to this is that occasionally there will be some legitimate IP's / emails / usernames that get caught in the net and denied access. For now though it seems to be a temporary solution to lighten the load. (At least till we get hit by the full force blasts again, which happened occasionally)

Once we are able to get a new firewall in place I think it should help resolve most the issues.

Eric Lyon

 

[twomoon]

I thought my multiple attempts to enter Namepros via dial-up connection caused all this situation, but good to know it's not me

Good luck to the NP staff and may the force be with you!

 

[hollywood]

Do I sense sarcasm?

Ray, if there was a sarcastic vibe in my earlier post, it was directed at my clients who said "we don't need that" while I maintained "Yes, you do".

There was no sarcasm intended to you, let me apologize if it may have sounded that way.

You had shared good info, but also within that post stated that passwords were "secure" in the currnt NP implementation. That statement appeared incorrect to me since you overlooked the client-server transit and the need for SSL.

But since you have more experience than me, I asked.

When you answered, I did feel better. And I tried to further the good information:

People should use SSL on their websites when they are maintaining personal customer data.

Even if the client says "I don't need it" its the web developer or admin's responsibility to say "yes, you do". In my honest opinion.



P.S. I have not experienced a DB conn error in the past two days, so I think we all should take our hats off and thank Eric for his efforts to mitigate this serious availability issue. Eric, you are the man! :kickass:

P.P.S. And just to share, I actually made a small domain purchase here at NP yesterday - something I have not done for over two years If the system were still failing, if Eric were not working on it, if fellow NP'ers were not helping out...then I would not have felt confident enough to buy a name here yesterday.

 

[silentg]

Are they still DDOSing NP? I am still getting the database errors.

 

[RazorNF]

Its becoming quite aggrivating, especially while in the middle replying or posting, and hitting preview or submit to get one of those errors.

 

[trainhappy]

I was wondering why I couldnt get to the site. Anyways, keep us updated.

 

[hollywood]

Wish there was a way to just shotgun all the ddos bots with one blast

Try rejecting at the router, not once they've reached the box? Good luck Eric - hollywood.

 

[Archangel]

These attacks aren't new to NP. We've had them over the years. Here's to hoping for a smooth fix.

 

[clicktechs]

Sorry you guys are under attack. I certainly know you are trying to fix as hard as possible. Good luck or whatever the right saying would be on this.

Spam, bots and few other things can make the net a drag at times.

 

[Eric Lyon]

Eric,

Another way would be to make a script to automatically ban an ip after x amount of queries per seconds, that way the server would block the ips that hit 100 queries per second, or something along those lines

Another good suggestion to look into.

Thanks

 

[snowbird]

well said snowbird and thanks for the ideas. That basically summed it up. early today we opened up 200 more ports just to see what the bots would do & sure enough, regardless how many ports we opened, they filled them up within 60 seconds.

I've re-enabled a few features that were collecting dust over the years that query a central spam database multiple vb users contribute to and initiated the auto blocking sequence again for known spam sources. It seems to be trimming down lots of the manual workload and leaving me with a more manageable amount of bots that slip through the cracks (never reported before).

The downside to this is that occasionally there will be some legitimate IP's / emails / usernames that get caught in the net and denied access. For now though it seems to be a temporary solution to lighten the load. (At least till we get hit by the full force blasts again, which happened occasionally)

Once we are able to get a new firewall in place I think it should help resolve most the issues.

Eric Lyon

Thanks for the update. I hear Cisco Guard is a pretty good firewall that also helps assist in managing and reducing DDOS attacks.

 

[earthgld]

Until i post this, still i often get the database error page. So the DDOS is the cause of all this, hope the team can get rid off the attackers. Good luck.

 

[Arief WN]

... For now I'll try to keep manually heading them off at the pass to keep the server loads down enough that we have fewer outages (fingers crossed), well, unless I'm sleeping, then it might build back up again till I wake up (gets some fresh coffee ready) ...

Eric, I think you need some good wingmen to back you up...

 

[Kamikaze]

Good luck, Eric!

 

[south]

configserver:

http://www.configserver.com/cp/csf.html

 

[Eric Lyon]

Well they must be after your database so you could ask users to change their passwords in case they have the same ones on their registrar accounts. And also secure the database if it is not already.

It's not that kind of attack at all. It's a botnet attack from spam bots trying to crash our servers.

Note: At NO TIME has ANYONE'S personal information been compromised!!!!

 

[liquidcherry]

just sue DNF for the attacks lol

cheers

liquid

disclaimer: in no way did i assume that DNF started those attacks, i just tried to be funny!!!!!!!!!!

 

[NetOctave]

Any way you could serve a cached / static version of the site ? Agreed people cannot post, but something that will keep us engaged ...
Specially I miss the Godaddy coupons... I end up getting it from Google cache and its sometimes a day or 2 old..