DataBase Errors

Thanks for the update Eric. I was wondering what the issue was for the last 12+ hours. I've issued an alert to the community on my blog and asked anyone with information to email me.

Cheers

Are they still DDOSing NP? I am still getting the database errors.

Its becoming quite aggrivating, especially while in the middle replying or posting, and hitting preview or submit to get one of those errors.

silentg said:

Are they still DDOSing NP? I am still getting the database errors.

Click to expand...

Yes, We have narrowed it down to Bot attacks and are still working on a solution.

RazorNF said:

Its becoming quite aggrivating, especially while in the middle replying or posting, and hitting preview or submit to get one of those errors.

Click to expand...

I agree, it's very annoying. Hopefully we can get this resolved sooner than later.

Sorry for the inconvenience,

Eric Lyon

Good luck Eric

Wish there was a way to just shotgun all the ddos bots with one blast

I was wondering why I couldnt get to the site. Anyways, keep us updated.

Bannen said:

Wish there was a way to just shotgun all the ddos bots with one blast

Click to expand...

Try rejecting at the router, not once they've reached the box? Good luck Eric - hollywood.

hollywood said:

Try rejecting at the router, not once they've reached the box? Good luck Eric - hollywood.

Click to expand...

We have plans later to put everything behind a new firewall which should rectify the situation completely. For now I'll try to keep manually heading them off at the pass to keep the server loads down enough that we have fewer outages (fingers crossed), well, unless I'm sleeping, then it might build back up again till I wake up (gets some fresh coffee ready).

I'll look into the router option a bit to see what's available.

Thanks,

Eric Lyon

These attacks aren't new to NP. We've had them over the years. Here's to hoping for a smooth fix.

Sorry you guys are under attack. I certainly know you are trying to fix as hard as possible. Good luck or whatever the right saying would be on this.

Spam, bots and few other things can make the net a drag at times.

Eric_Lyon said:

We have plans later to put everything behind a new firewall which should rectify the situation completely. For now I'll try to keep manually heading them off at the pass to keep the server loads down enough that we have fewer outages (fingers crossed), well, unless I'm sleeping, then it might build back up again till I wake up (gets some fresh coffee ready).

I'll look into the router option a bit to see what's available.

Thanks,

Eric Lyon

Click to expand...

Eric,

Another way would be to make a script to automatically ban an ip after x amount of queries per seconds. For example; the server would block the ips that hit 100 queries (depending on the excessive amount of queries) per second.

Nathan said:

Eric,

Another way would be to make a script to automatically ban an ip after x amount of queries per seconds, that way the server would block the ips that hit 100 queries per second, or something along those lines

Click to expand...

Another good suggestion to look into.

Thanks

cobo_836 said:

Means that these days database error may occur?Do not be surprised to encounter such a situation, is it?

Click to expand...

It is not out of the ordinary for this to happen considering a forum like this that is database driven is using dynamic content that requires a database connection. Since dynamic content is more load intensive than loading static content, there are limitations placed on how many database connections there can be allowed at one time.

Ironically Eric, I saw you mention Chinese IPs and I recall about a couple weeks ago one of my servers being hit with failed brute force attempts on my ssh port from IPs in China. Apparently that is becoming a pretty popular location for obtaining a proxy.

As for DDOS, unfortunately it is an attack that is hard to prevent these days. There is no sure thing protection against it and the best way to mitigate them is to have lots and lots of bandwidth access to the backbone. DDOS is not really the problem here, the problem is all the botnets that are out there generating these attacks and with all the broadband internet connections made available to so many out there, it is becoming easier and easier to perform these kinds of attacks. Another method to help mitigate these problems is using Anycast which takes the server IP for namepros and routes it to several different servers hosted at several different hosts. In this method, people who hit the server's IP will be routed to the best available server based upon resource usage and location. This would help distribute the traffic through several networks vs. just 1.

Those who say to create a script to just ban the offending IPs, that can work for smaller DDOS attacks, but for serious attacks, the network is still being hit regardless. But that might be a good course of action at this time however since we are in fact seeing a database error which is a good sign. We are reaching the web server. There just appears to be too many database connections probably which is preventing some people from being able to access the dynamic content.

well said snowbird and thanks for the ideas. That basically summed it up. early today we opened up 200 more ports just to see what the bots would do & sure enough, regardless how many ports we opened, they filled them up within 60 seconds.

I've re-enabled a few features that were collecting dust over the years that query a central spam database multiple vb users contribute to and initiated the auto blocking sequence again for known spam sources. It seems to be trimming down lots of the manual workload and leaving me with a more manageable amount of bots that slip through the cracks (never reported before).

The downside to this is that occasionally there will be some legitimate IP's / emails / usernames that get caught in the net and denied access. For now though it seems to be a temporary solution to lighten the load. (At least till we get hit by the full force blasts again, which happened occasionally)

Once we are able to get a new firewall in place I think it should help resolve most the issues.

Eric Lyon

Eric_Lyon said:

well said snowbird and thanks for the ideas. That basically summed it up. early today we opened up 200 more ports just to see what the bots would do & sure enough, regardless how many ports we opened, they filled them up within 60 seconds.

I've re-enabled a few features that were collecting dust over the years that query a central spam database multiple vb users contribute to and initiated the auto blocking sequence again for known spam sources. It seems to be trimming down lots of the manual workload and leaving me with a more manageable amount of bots that slip through the cracks (never reported before).

The downside to this is that occasionally there will be some legitimate IP's / emails / usernames that get caught in the net and denied access. For now though it seems to be a temporary solution to lighten the load. (At least till we get hit by the full force blasts again, which happened occasionally)

Once we are able to get a new firewall in place I think it should help resolve most the issues.

Eric Lyon

Click to expand...

Thanks for the update. I hear Cisco Guard is a pretty good firewall that also helps assist in managing and reducing DDOS attacks.

Until i post this, still i often get the database error page. So the DDOS is the cause of all this, hope the team can get rid off the attackers. Good luck.

Eric_Lyon said:

... For now I'll try to keep manually heading them off at the pass to keep the server loads down enough that we have fewer outages (fingers crossed), well, unless I'm sleeping, then it might build back up again till I wake up (gets some fresh coffee ready) ...

Click to expand...

Eric, I think you need some good wingmen to back you up...

Those jobless scumbags doin this should be put in jail.

Good luck, Eric!

configserver:

http://www.configserver.com/cp/csf.html

Well they must be after your database so you could ask users to change their passwords in case they have the same ones on their registrar accounts. And also secure the database if it is not already.

steelbangle said:

Well they must be after your database so you could ask users to change their passwords in case they have the same ones on their registrar accounts. And also secure the database if it is not already.

Click to expand...

It's not that kind of attack at all. It's a botnet attack from spam bots trying to crash our servers.

Note: At NO TIME has ANYONE'S personal information been compromised!!!!

steelbangle said:

Well they must be after your database so you could ask users to change their passwords in case they have the same ones on their registrar accounts. And also secure the database if it is not already.

Click to expand...

The type of attack you are referring to Is a SQL injection where a hacker exploits an unsecure database in order to harvest information. This attack could also be used to bring down a particular web site by destroying data and tables, but since members have access at times and others don't and the database is completely intact, I do not believe it is that method.

What namepros is experiencing is a DoS attack, or Denial of Service. Which generally happens is a hacker uses a trojan to generate a gruesome amount of request from average household computers without the owners knowing. Generally, it is successful as the server is unable to process the large packet size and rate and will eventually time out or hang.

DoS attacks have been used against many reputable Fortune 500 companies to include google. There is no one way to prevent it with completely denoting access to the average joe.

In this case, which is generally common with forums, a spam bot is submitting hundreds of requests to create accounts and posts over and over and is simply not able to make it past the human validation.

To mitigate this issue, I recommend instating a load balancer with a dedicated SQL server. Increase the number of maximum connections in correlation with the processor, and use HTACCESS and iptables to block the current known block of ips. Install and update network tools such as APF and DoS deflate as previously mentioned. DoS Deflate will automatically detect DoS attacks and deflect them at the system level by adding them into the Iptables and not allowing the processor to accept requests. Also make sure the Linux kernel is up to date . A hardware firewall or NAT would be extemely helpful aswell.

I also suggest blocking namepros from the particular country of origin for a short time as the request will return back empty and eventually stop. I understand that this may create issues but it is the for the betterment of the community and would only be a short time. A member could access NP from an established VPN or proxy during this time.

Use this command in the kernel to find ip ranges with abnormal connections

#netstat -anp|grep tcp|awk ‘{print $5}’| cut -d : -f1|sort|uniq -c|sort -n

Click to expand...

To address the issue that was brought up regarding passwords, I'm pretty sure that vbulletin uses bcrypt (blowfish encrypt) with a salt and a time deflect. For those that don't know bcrypt it is one of the strongest commercial grade encryption methods as it prevents the use of rainbow tables by using a salt and is time generated (key is generated at a certain rate in the salt which prevents Bruteforce) It's way more secure than md5, mcrypt, and the standard MySQL encrypting function.

That being said, unless namepros stores the passwords in plain text which I highly doubt under the worst circumstances our passwords are secure, even if using an older method.

Still one should always take precaution, regularly change passwords, and never link multiple accounts to one email address

Ray

Source: 5+ YR server and website administration.

just sue DNF for the attacks lol

cheers

liquid

disclaimer: in no way did i assume that DNF started those attacks, i just tried to be funny!!!!!!!!!!