I use Verizon FIOS as my home ISP. I like it , it’s very reliable and Verizon is very responsive to my problems. Recently, when I reply to emails, their server, outgoing.verizon.net won’t send the email. It rejects it with a 5.7.1 notice and calls it spam. I can sympathize why do this. Verizon like every other ISP, doesn’t fully trust their customer base and SPAM is a scourge on the world’s email infrastructure.
Why does this happen? Apparently they generate a signature and if this signature looks like a spam payload they will reject the email.
So why don’t I use someone else to send email? On all of the home ISPs port 25 is blocked except your local authorized email server. On Verizon port 25 is not used. Their SMTP port is encrypted and appears on port 465. Of course if you have a vpn tunnel to somewhere else (i.e. work) you could use their mail server but I don’t have that luxury so I am stuck with what Verizon provides. This is not an exhaustive list. There are a lot of ports that could be used but the common ones are 25, 465 and 587.
<Sigh> However </Sigh> I just wish Verizon would give me the benefit of the doubt. I am using an encrypted connection to their SMTP server with a Verizon assigned username/password. They know who is sending the emails. (Email headers can be easily forged but the encrypted connection is harder to hack). I wouldn’t mind being in their good graces until someone complains about me.
At least they have a work around. Namely you send the email to: firstname.lastname@example.org
as the *only* recipient and wait 30min-1hour and they will put this email in a whitelist. You can resend the email after than and it won’t get bounced.
While watching Elementary I saw a reference to a obscure programming language called “Malbolge“. Malbolge or Malebolge is the eighth circle of Dante’s hell. Turns out , according to the program ,Malboge is a language “designed to be as difficult to program in as possible”. I first thought that was pure fiction but it is a real programming language.
No really! It uses ternary logic rather than binary.
Unfortunately some of the programs that I had to maintain were trying to emulate Malboge. I suspect that I am not alone. Most programmers who had to maintain someone else’s code would say “This guy is/was an idiot. I need to rewrite the program from scratch”.
We don’t know why my wife’s website was DDOS attacked. The site is an Artist’s website which appeals to a select group of viewers. Until a few weeks ago this site was attracting about 50 visitors per day. Then we got a
503 Service Unavailable
The server is currently unable to handle the request due to a temporary overloading or maintenance of the server.
I figured that there was something wrong at our hosting provider. But no that was not the issue. I tried logging into the site using ssh. First I couldn’t ssh into my account. I got “could not fork processes” serveral times before I could login. Then when I tried to do a ps I got “no more processes”. Again I had to repeat this several times before I got a ps output. I immediately noticed that I had a ton of httpd daemon processes. Also I had a ton of email processes. Hoping to reduce the number of processes I had my hosting provider turn off email for this account but that did no good. Eventually my hosting provider turned off the website since my wife’s website was affecting the other servers. That’s when I realized that my hosting provider was not part of the solution, they were part of the problem.
‘There is no more neutrality in the world. You either have to be part of the solution, or you’re going to be part of the problem.’ – Eldridge Cleaver
My hosting provider for this site was Verio and we used to be very happy with them. Many years ago we had hosting with a company called li.net. Then Verio bought them but things went well for a long time after that. We were always able to contact their systems Admins directly so if there was a problem we could solve it quickly with just a bit of dialogue.
Until recently.. It started when we had to register a new shared hosting site with them. A procedure that should have taken minutes took a couple of days. Hmmmm.
That’s what the customer service people at Verio told us we had to do. We went from 50 page views a day to “You are running out of processes, you need to upgrade your plan”. They originally told us that we were receiving too much email. So I told them to discontinue the email plan. That fixed nothing and we had no email. Then Verio customer service said “You are getting too many page hits! you need to upgrade your plan!” . I asked for the access_logs and error_logs for the site and got something like this:
188.8.131.52 - - [18/Aug/2013:21:00:36 -0400] "GET /website/wp-content/gallery/flowers-in-the-garden/2012-sunflowers-w-asters.jpg HTTP/1 .1" 304 - "http://maryahernartist.com/nextgen-galleria-gallery/668" "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0"
184.108.40.206 - - [18/Aug/2013:21:01:37 -0400] "POST / HTTP/1.1" 200 36072 "-" "Mozilla/4.0 220.127.116.11 - - [18/Aug/2013:21:06:55 -0400] "POST / HTTP/1.1" 200 35865 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
18.104.22.168 - - [18/Aug/2013:21:07:19 -0400] "POST / HTTP/1.1" 200 36111 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
22.214.171.124 - - [18/Aug/2013:21:07:20 -0400] "POST / HTTP/1.1" 200 35999 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
126.96.36.199 - - [18/Aug/2013:21:07:22 -0400] "POST / HTTP/1.1" 200 36048 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
188.8.131.52 - - [18/Aug/2013:21:07:33 -0400] "POST / HTTP/1.1" 200 36048 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
184.108.40.206 - - [18/Aug/2013:21:07:51 -0400] "POST / HTTP/1.1" 200 35928 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
220.127.116.11 - - [18/Aug/2013:21:07:53 -0400] "POST / HTTP/1.1" 200 36016 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
18.104.22.168 – – [18/Aug/2013:21:08:04 -0400] “POST / HTTP/1.1” 200 36055 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
22.214.171.124 – – [18/Aug/2013:21:08:01 -0400] “POST / HTTP/1.1” 200 36048 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
126.96.36.199 – – [18/Aug/2013:21:08:23 -0400] “POST / HTTP/1.1” 200 36055 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
188.8.131.52 – – [18/Aug/2013:21:08:37 -0400] “POST / HTTP/1.1” 200 36072 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
184.108.40.206 – – [18/Aug/2013:21:08:38 -0400] “POST / HTTP/1.1” 200 36111 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
220.127.116.11 – – [18/Aug/2013:21:08:45 -0400] “POST / HTTP/1.1” 200 36062 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
18.104.22.168 – – [18/Aug/2013:21:08:46 -0400] “POST / HTTP/1.1” 200 36016 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
22.214.171.124 – – [18/Aug/2013:21:07:27 -0400] “POST / HTTP/1.1” 200 36072 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
127.0.0.1 – – [18/Aug/2013:21:08:57 -0400] “OPTIONS * HTTP/1.0” 200 – “-” “Apache/2.2.15 (CentOS) (internal dummy connection)”
126.96.36.199 – – [18/Aug/2013:21:09:12 -0400] “POST / HTTP/1.1” 200 36072 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
188.8.131.52 – – [18/Aug/2013:21:09:18 -0400] “POST / HTTP/1.1” 200 36065 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
184.108.40.206 – – [18/Aug/2013:21:09:20 -0400] “POST / HTTP/1.1” 200 36062 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
And so on ad-infinitum… Boy those ie 6.0 Windows XP boxes sure like us! Thus ladies and gentlemen is what a HTTP Flood Attack looks like.
The next suggestion from Verio was to upgrade to WordPress 3.6. We were using last weeks version 3.5.2. This is difficult to do via command line but I did it. We also had to upgrade all the plugins. Then we turned the site back live and guess what? It was still falling over.
I had gotten nowhere with Verio. And looking back I don’t know what they could have done anyway. The main issue I have with them is that they firewalled their Systems Admins from their customer base. As a customer you can not have a conversation with an SA, only their Customer Service. These people are not empowered to fix any problems, they can only relay the issues upstream and relay the answers downstream.
It was time to look elsewhere. I started looking into Virtual Private Servers (VPS). The one Verio offers costs $30. Oh boy they want us to pay $30 as opposed to the $10 that we were currently paying. I searched and found a VPS provider in New Jersey. www.interserver.com I ordered the most basic service and wound up paying $6 as opposed to $10. To be truthful , the $6 offers similar performance to the shared server, which quite frankly is nothing to write home about. However, you can upgrade the slices. This doubles the memory allocation, web bandwidth, disk allocation per slice. E.G.
And so on.. I chose OpenVZ rather than KVM for 2 reasons:
1) Cheaper ie. $6 vs $8 per slice
2) (Slightly) better performance per slice. The disadvantage is that you cannot change the kernel and there is less isolation. I am using the guest OS , same as everyone else, so I did not think that kernel mods were necessary.
Interserver also sells upgrades ie. SSDs, which are $4.00 per slice and a control panel . They have two types of Control Panels , Cpanel and DirectAdmin. DirectAdmin costs $8 whist Cpanel costs $10. Note they require at least two slices to run. Cpanel or Control panel does more and costs more. I think that Interserver has to pay a 3rd party which explains why they are relatively expensive. For those who know me I am a command line guy anyway so I have no use for these Control Panels. The SSDs seem intriguing . I might spring for one to see if it makes any difference. (It should, especially with only 512 MB of memory).
I think this is a great deal. Most other VPSs start at $30 so this is a tremendous bargain. Also I am currently on a $.01 promotion so the first month only costs me $.01!!
So I bought a single slice VPS and set it up. BTW the purchase process is very easy! And setting up the Centos server is easy.
N.B. This is not Linux centric. You can order Windows (2008R2 for example). However the price goes to $11 per slice.
One smart thing that we did was register the domains with a separate registrar. We use Network Solutions. Which , I believe, was the first Internet Registrar. I like them well enough. But then I use an ATT iPhone and I like that well enough . So at the NS control panel I re-pointed the DNS servers from Verio to Network Solutions. And then I pointed the Address records to my new server at interserver.com. And guess what, the Apache web servers fell over, probably even quicker than the one at Verio.
One of the issues for the DDOS attack was that we are on a shared hosting plan. This has the advantage of being Cheap ($). But you are sharing the same server with several others. Old(er) developers among you might remember a time when programming was done via terminals , connected to a centralized system. These systems were usually Unix Boxes and were known as multi-user systems. Windows Systems don’t have the concept of multi-user even on so-called Windows Server boxes. This is because computing horsepower is so cheap. Everyone can have their own computer. But Unix and Linux systems still have this feature even to this day.
What does this have to do with shared hosting? It turns out that web-servers can serve up different websites based upon how they are accessed. Apache calls this Virtual Host. Nginx uses server blocks. I’m sure that IIS has a way of doing this as well. For Example if www.domain1.com points to 220.127.116.11 and www.domain2.com points to the same IP Address. If Virtual Hosts is properly setup then domain1’s webpages will be different than domain2’s webpages. The users of domain1 will think that they are on a completely different site than domain2 and in essence they are.
Here is a better explanation:
On my shared hosting account I could log into my account but I was just a user. I could not see the access and error logs. This can be provided for me but Verio didn’t bother.
So what is a VPS? A VPS (Virtual Private Server) is a virtual host in the cloud. The VPS provides an operating system with some software pre-loaded. There is a control panel where I can stop/start the server. Re-install the OS if I wanted etc. They provide ssh access to it and you can login as root. In short you own the box. I chose a local VPS provider interserver.com, So far they are excellent. I chose the cheapest plan for $6.00. This is even cheaper than Verio’s shared hosting. You can get a site based in New Jersey unlike Colorado where the Verio servers are located. Since I am a Linux SA I found it easy to setup Apache. The great thing is that I fully controlled the box so I can access the logs, configure Apache and security the way I saw fit.
It turned out that I needed a VPS to fix my DDOS problem.
At this point Mary’s web server had been down for over a week. I knew that we had a DDOS attack of some sort but I didn’t know what to do. After looking at this post in the InterServer Forums I got some ideas.
Basically you have a few options.
1) You can detect incoming bad guys and null route them. This means that their IP is blocked from future accesses to your server. This would be good if the Attacks were not distributed. I.E. they were coming from a few hosts. However, I tried to null route using a shell script and got to over 700 bad Ips with no end in site. And eventually the system locked up.
2) You can a more efficient server. Quags the Admin recommended using nginx and a more efficient firewall. I configured and ran nginx but this didn’t work for me.
3) You can get external help ie. Use a reverse proxy. Quags recommended www.cloudflare.com. Since I was at the end of my rope I decided to use them. Turns out they have a free package. (If it’s free then it’s for me!). You can set the cloudflare Security Settings to “I’m under a DDOS attack”. I also had to set my firewall to reject all Ips except those from cloudflare. Then miraculously the website started functioning. However, each visitor got this interstitial page:
I ran a short survey on Facebook to see if this 5 second delay was a show stopper: Yes it seemed that for some it indeed was a show stopper. I would also suspect that most casual web users would see this and bypass it and go somewhere else. My facebook friends were warned that there was a 5 second delay so they were more understanding than the average user.
Somehow I stumbled upon a security module for Apache. It is called strangely enough mod_security. It is a snap to install .
yum install mod_security <cr>
yum install mod_security_crs <cr>
Be aware that mod_security by itself does no work. You need the filters. They live in
I used all the filters except the one for sql injection. And then I turned the security setting in cloudflare from “I’m under attack” to “high”. Voilla Now the site “just works.” Oh boy!
So what did I learn.
1) You can solve at least some DDOS issues for no $$$. So far the solution costs me $.01. Next month it will be $6.00 less than what I originally paid.
2) Shared hosting seems useless for DDOS attacks. I had to change firewall and Apache settings. I don’t see how this would work in a shared environment.
3) You have to monitor your hosting provider to see if they changed their policies for the worse. Apparently mine did.