I have something VERY exciting to tell you about, and instead of telling a 2500-word story first and saving the punch line, I’m going to tell you the news first. I’m still going to tell the 2000-word story but you just won’t have to wait for the dramatic reveal. Here’s the announcement – podfeet.com is WICKED FAST. We have finally solved the problems that were plaguing it, and it’s now faster than I ever thought possible. My hope was that we’d get it down to maybe 2-second page loads, but it’s way way faster than that now. Ok, with the fun part out of the way, how did we fix it? We’ll start with a tiny bit of review.
A few weeks ago I talked about how I was working hard to try to figure out why podfeet was so slow. Podfeet hasn’t been the snappiest website for a long time, but it had really hit a wall in performance, making it nearly unusable. I threw money at the problem first by doubling the number of virtual CPUs, and while that did get the server out of its temporary jam, very quickly the percentage CPU usage started to climb again. We were out of 40-second page load times, but we were still seeing hours at a time where the CPU was just under 100% usage.
Cloudflare
Bart’s next idea, as I explained last time, was to move the database to a dedicated server, and that’s when our hero, William (Bill) Reveal joined the party to figure out how to fix that pesky encoding problem. As I reported, things got faster after we successfully moved the database, but still, we had those pesky 98% high load periods for extended periods of time.
Bart’s next idea was to move podfeet.com under the protection of Cloudflare. They have a free service where they manage the DNS records (those are the records that allow you to type in a URL and it gets converted to the correct IP address.) Cloudflare specialize in protecting those domains under their wing against denial of service attacks. We didn’t think my server was under a DDoS because the traffic numbers didn’t account for that, but we knew it wouldn’t hurt. It didn’t hurt, but it also didn’t help.
Moving podfeet.com to a VPC
At this point, my web server and my database were on separate virtual machines, and both were hosted by DigitalOcean. But Bill noted that the traffic between them was on the open Internet which isn’t recommended from a security standpoint if you can help it.
In the DigitalOcean service, if you create a new Droplet today (their name for a virtual machine), they put it in what they call a Virtual Private Cloud, or VPC. VPC is a logically isolated network for cloud resources. The database server was created in their VPC, but the original web server was built long enough ago that it was not. Bill suggested that the traffic would be more protected and hopefully the site would be served more quickly if both servers were on the VPC.
Luckily, it’s very easy to move a droplet into the VPC with DigitalOcean. You simply press a button to create a snapshot of your server, turn off the original server, and turn on the snapshot in the VPC. I bet you’ll be shocked to learn that this method did not work for us. The problem was something called cPanel.
I’m what my friend Linda would call “a reasonably bright girl”, but I’m definitely still learning when it comes to managing my servers myself. Bart has been slowly but surely teaching me, by having me do all the typing and configuring while he directs. But to be honest, a great deal of it often flies over my head. One of those fly-over-my-head parts was a few years ago when he suggested I use something called cPanel to help me manage my site.
Until two weeks ago, I thought cPanel was just a nifty interface to get to things like phpMyAdmin to play with my database, add certificates, and update things. Evidently cPanel is much much more, and it was actually part of the structural fabric of my web server. I know that now, because the existence of cPanel in my environment meant it was not possible to put the snapshot of podfeet.com into the VPC.
Bill and I did a lot of searches about how to extricate cPanel from the server and they all said, “Don’t even try, you must burn it to the ground and start over.” So that’s what we did.
PHP-CGI vs PHP-FPM
Bill was very much in favor of this move for a more fundamental reason. I’ve mentioned before that web servers are often on what’s called the LAMP (or WAMP) stack. The acronym stands for Linux Apache MySQL and PHP. That’s the operating system, the web server software, the database, and the programming language. When Bill and I were done, my web server was no longer on the LAMP stack. I think the changes and debates along the way were really interesting.
As I said, PHP is the programming language in LAMP, but it turns out there are two versions of PHP, which I didn’t know before. I promise I won’t get too deep into this part (mostly because my understanding is fairly limited) but it may actually be the most significant part of the story.
PHP is a way to run a server-side script when an HTTP request comes in. The two versions of PHP are PHP-CGI and PHP-FPM. PHP-FPM is the “fast” version of PHP-CGI. As Bill explained it to me, PHP-CGI is better for low-traffic sites but PHP-FPM (FastCGI Process Manager) is significantly more efficient for high-traffic sites. (Learn more at basezap.com/…)
I had to ask Bill, “Is my site high or low traffic?” I know podfeet.com isn’t the New York Times, but it’s also not Sally’s Left-Handed Wingnut Supply Store either. He did some research into my traffic and declared podfeet.com officially “medium”. He said I should be congratulated. I was so proud.
Bill said that if we’re going to rebuild podfeet.com from scratch, we should do it using PHP-FPM. He based his theory on the fact that the only thing that seemed to be churning on my server when these CPU bottlenecks were happening was PHP-CGI. PHP-FPM it is.
Building podfeet.com From Bare Metal
I didn’t really appreciate what we were about to undertake in building podfeet.com from bare metal. While it was a ton of work, it was fascinating and I learned so much from Bill. Like Bart, Bill directed while I did all of the typing. I’m not even going to pretend that I understood everything we did, and I definitely couldn’t replicate 90% of what we did, but I got a better appreciation for what it means to build a server from scratch.
The first thing you have to do is decide on an operating system. This first decision created an interesting discussion between Bart, Bill, and me. Bart likes CentOS and it’s what he manages with his university work hat on, but if it was all up to Bill he would have gone with Ubuntu. CentOS is more of an enterprise-level OS with lots of security features built in, but being an enterprise OS means it moves more slowly so it’s actually using a fairly old version of PHP. Bart, Bill and I had to discuss which way to go.
My worry at this point was that I would be caught between the two people doing so much work to help me. Nerds can get into holy wars but that wasn’t going to happen with Bill and Bart. Because they’re both fine, intelligent people with good hearts, we quickly decided to go with Bart’s choice of CentOS because we know he’ll be working with me for the long haul. I suspect Bill will be too, but we had to flip a coin and the toss went to CentOS.
The second decision was which web server software to go with. I said that the A in LAMP was for Apache, but Apache isn’t the only option. Bart prefers something called NGINX. I remember a hundred years ago he stood up a tiny IRC chat server for me and it used NGINX, but I didn’t really know what that meant back then. Bart sees NGINX as much easier to manage than Apache, but of course Bill was much more comfortable with Apache. The coin toss went to Bart again, and we went with NGINX.
Remember I said my webserver isn’t on the LAMP stack anymore? The M is gone altogether because MySQL isn’t even running on the web server, as the database is on a separate, dedicated server. That would leave me with LAP, but the A is gone too because I’m not using Apache as the web server software, I’m using NGINX. That means I have an LNP server. Doesn’t quite have the same ring to it, does it? Can’t even pronounce it.
Another thing I didn’t appreciate before we went into this exercise is what doesn’t exist when you stand up a virtual machine with JUST an operating system. There was nothing there. I mean not even a user account for me! We even had to install tools in order to install tools.
The good news for you is that I can’t remember everything we did and probably couldn’t explain a lot of it. Once we had NGINX functioning as a web server, it was actually a pretty easy task to copy over all of the WordPress files to the new server and point the Cloudflare DNS at the new server.
There was another piece to figure out, and it’s not completely working just yet. You know how I tell you to go to podfeet.com/patreon to help support the show, but if you go there it actually takes you to patreon.com? That works because of what’s called a redirect, and in Apache, it’s stored in a text file called .htaccess.
Donald Burr actually helped me build my .htaccess file years ago, and every time I think of something new for which I want to give you an easy link, I just copy a line Donald wrote and change it for the new redirect. I love these redirects because if at some point there’s a better way to support the show than Patreon, I can keep podfeet.com/patreon and redirect it to the new service. I also use these for internal links, like podfeet.com/live actually goes to https://www.podfeet.com/blog/nosillacast-live/ which is way too hard to remember.
But NGINX doesn’t use .htaccess, it uses a .conf file inside the NGINX directory. The syntax is completely different, and I don’t understand it very well at all. It was unfamiliar to Bill as well, and we got most of it working but even with his help, we managed to make an infinite loop of redirects within podfeet.com at one point. Since without Patreon and PayPal I don’t get paid, I wasn’t willing to move the server over until we had at least those two links functioning.
Eventually, we got most of the redirects working so we flipped the switch to move to the new, built-from-scratch server.
How Fast Is It?
I told you it’s fast now, and I even said wicked fast. I can prove that with real numbers. When all this mess started, I installed a plugin for WordPress called Query Monitor. We installed it because it would report on errors and warnings that we hoped would point at a root cause. It didn’t find anything wrong which was disappointing but actually good news. But Query Monitor also has a little display that shows when you’re logged into WordPress to tell you how fast the pages are loading.
After I doubled the number of CPUs, the page loads went from as bad as 40 seconds down to 6 seconds. On occasion when podfeet.com was in a good mood, it was down to 2 seconds to paint a page. I know that sounds great, but 2 seconds is actually really annoying.
After Bill and I built the webserver from scratch running NGINX and PHP-FPM with a dedicated database server and using Cloudflare, I am happy to report that most pages come up in a quarter of a second! Seriously, a 10X improvement in speed.
I find it interesting that we still don’t actually know why the original server was churning the CPUs so hard. We know the database move didn’t fix it and neither did going behind Cloudflare. We’re pretty sure nothing was attacking it from the outside, but something was clearly going on inside the machine. Was it PHP-CGI? Was it Apache? Was it nearly 17 years of cruft and just time to do a nuke and pave? We’ll never know. The one thing we do know is the speed of podfeet.com makes me deliriously happy.
It made me happy when I created my diagram about the different M1 options and I knew if it got popular people would get it quickly (which helps it get more popular). It also makes both Steve and me positively joyous because we’re the ones who go into the WordPress interface all the time which took FOREVER until now. I love it so much that I just keep going to page after page on my website to watch it blink and load. Of all the comments about how fast the site is now, I think my latest favorite is what Allister Jenks said in Slack: “It’s disconcertingly fast.”
Bill is My Hero
Another super fun side effect of this work was that Bill slowly and gently improved my skills using the vi editor. He didn’t make me use it but I said I wanted to get better at it. By the end of two weeks of working together, he started noticing that I was getting faster at anticipating what he was going to tell me to do and becoming more natural in using the tools of managing a server.
I simply can’t thank Bill enough for all of his help. I said he did a lot before, but what he did in the last couple of weeks and the time he spent was beyond generous. He was fun, he taught me many things, we laughed a lot (mostly geek jokes) and we were successful. I told him that Steve and I felt we should pay him for this time, but he didn’t agree. He said that he’s been listening to the NosillaCast for nearly 15 years and that he’s never contributed anything. He was happy that he was finally able to contribute in a meaningful way.
Bill is my hero.
Please go to podfeet.com and push all the links and rejoice with us at how fast it is!