Morris "Mojo" Jones

Mojo's blog banner

Tag: Apache

  • Moving Whiteoaks.com — Building a mail server

    Don’t do this. This is stupid. Running a mail server is a pain in the ass.

    If your startup or family or organization needs a domain and email, just point your domain at Google and use their great mail, calendar, and shared documents tools. Avoid all the battles with spammers, black lists, hackers and security updates.

    So why would I do this? Maybe because I always have — I’ve had a server connected to the net since 1995. I enjoy being master of my domain(s). In a perverse way it’s a challenging hobby as well.

    My server for whiteoaks.com lives in the linen closet at the top of my staircase. It really doesn’t get enough cooling, and when the processor gets too busy it complains about the heat. The version of Linux on it is old and long unsupported. I’ve been having to build package upgrades from source — like gcc, apache, php (4 and 5), and mysql.

    I know I’ve needed to build a new server, but I’ve been dreading putting together a machine and painstakingly migrating my services, users and domains to a new environment.

    Meanwhile Jane and I have discovered the joys of streaming Netflix and Amazon to the home theater. That means sharing my server’s DSL line to download TV and movies — not great for either service.

    I pay Verizon about $95 per month for a commercial DSL account with 3 mbps download and 768 kbps upload. For a server, 768 kbps is pretty slow. Moving to the cloud would give my server cloud-sized speed as well, and let me switch to a consumer internet account at home — cheaper and probably faster.

    So it’s really long past time that I moved out of my linen closet and into the cloud.

    Much techie stuff follows. Feel free to skip down!

    I spent several hours pricing out servers from Rackspace and others, trying to figure out what it would cost me. Right now I run everything — email, DNS, a dozen or so web sites — from one eight-year-old server with 1GB of ram and 40GB of disk.

    I wanted to upgrade my mail service to use Zimbra, and I knew it would require its own dedicated server with 1GB of ram. On Rackspace, that’s “flavor 3” and would cost me about $40 per month.

    Here’s why I wanted Zimbra: It’s available in a free open source community edition. It’s built on top of all the open source packages I was already using and familiar with: Postfix, Clamd, Spamassassin, MySQL. Zimbra provides a first-class web mail client that would enable my users (mostly family) to do things like change their password and set up filters to route mail into folders.

    Over my vacation, I did an experiment. (I did a lot of this using wi-fi on a Delta flight. A first-class seat makes for a delightful work environment!)

    I spun up a 1GB Rackspace server using Ubuntu 10.04LTS as the base. I set up a subdomain “rs.whiteoaks.com” using Rackspace’s excellent DNS support. I made an MX record to point to the new machine mail.rs.whiteoaks.com. Rackspace DNS support even let me set the reverse DNS to resolve mail.rs.whiteoaks.com to this server — essential on an email server!

    (Rackspace DNS is so good that I can easily drop my own BIND DNS server, with no regrets. Let them deal with DNS security!)

    On the new server I went through the Zimbra install script, interrupted only to install a couple of missing packages, and found myself a few minutes later with a genuine working email server that would send and receive email.

    As an experiment, I made an image of that server, destroyed the server, and created one from the image. It came up okay, but I couldn’t connect IMAP clients to it, such as my Android phone email. (I later realized that the self-signed SSL certificate created during the Zimbra install needed to be recreated for the new IP address.)

    Zimbra does not include a first-class mailing list manager. I host mailing lists for several astronomy clubs, OTASTRO, Bridgemojo and such — not huge lists, but as large as a few hundred members.

    The best non-commercial mailing list manager for many years has been Mailman, supported by the community at http://list.org. I’ve always used
    Mailman for my lists.

    This led to much teeth-gnashing as I started researching how to integrate Zimbra and Mailman.

    Zimbra really doesn’t like to have anything else running on its server. I understand this. It’s not just a control-freak thing, it means they can build and create an easily-installed turnkey box.

    This has led to two strategies for integrating Mailman and Zimbra: the cowboy strategy of running Mailman on the Zimbra server alongside Zimbra, and the officially supported strategy of running Mailman on its own separate server, with a Zimbra-provided API hook for better integration.

    I knew I was going to have another server for my web services, so I thought I would start out with the “separate server” strategy. Installing Mailman is simple enough, but then it has to be knitted carefully into an Apache web server as well as Postfix.

    This quickly began to seem like the more fragile and complex of the two solutions. It would require patching Mailman and Zimbra to support Zimbra’s API to manage all of the required mailing list aliases. It also will not support multiple domains, which makes that strategy dead on arrival.

    So back to the cowboy approach, knit Mailman into the Zimbra server, hopefully without making too much of an impact on Zimbra’s ecosystem.

    This took some doing, but was a good project for a Sunday while watching the final round of the Masters.  🙂

    I won’t get directly into the details here. I know there will be sysadmins curious about the problems I encountered and solved. None of the “how-tos” and forum threads covered every problem, and many of them were several years old. I owe it to the community to write up as much of my problems and solutions as I can. I promise to do that in a follow-up blog post.

    The final result is a thing of pride and beauty. Here’s a little of what I did.

    Mailman needs to be knitted into two services: Apache (web service) and Postfix (mail service). On Zimbra’s box, Zimbra owns both of these.

    The web service knit turned out to be reasonable and elegant. The solution I liked is to move the Zimbra server out of the way to another port (81 in my case).  Zimbra provides a local configuration flag to change that port that will be preserved in an upgrade. Then I installed a server-wide Apache package and configured it for Mailman in a virtual host. In another virtual host, I set up an http proxy that will forward to Zimbra’s apache instance on localhost port 81.

    With this setup, the Mailman web users will find the familiar Mailman applications at lists.rs.whiteoaks.com, and the Zimbra users will find their email applications at mail.rs.whiteoaks.com.

    The second knitting job requires patching Mailman into Zimbra’s owned and operated Postfix instance. There are only three spots that need to be patched in the Postfix main.cf configuration file, alias maps, virtual alias maps, and mydestination. Patching them works fine.

    But here’s the rub: every time Zimbra restarts Postfix it will rebuild main.cf from its configuration. This is annoying at first, and brilliant in retrospect. By setting Zimbra’s local configuration variables, the Mailman hooks will be preserved not only through restarts, but through package upgrades of Zimbra. (Details forthcoming, I promise.)

    A few days ago I began forwarding my main email inbox to the mail server on my Zimbra instance, and yesterday I migrated the mailing lists for Bridgemojo.  Both are working swimmingly.

    A clean untrained Spamassassin lets through a lot of marginal junk mail, but I was delighted to learn that clicking “Junk” in Thunderbird and moving spam to the Junk folder would automatically train Spamassassin.  Likewise finding good email in the Junk folder and clicking “Not Junk” would also train Spamassassin.

    I can now use “push email” from the Zimbra IMAP server on my Android phone. My old “wu” imapd servers could not support push along with a desktop email client without getting hopelessly tangled. The old servers are also developing a habit of hanging and locking up changes to the mailbox until I find the problem and kill it.

    I knew this would be the hardest part of migrating to the cloud. Moving the web services should be a piece of cake. One additional 512MB server should handle those easily, especially with no mail services chewing up memory.

    Before publishing this, I moved mojo.whiteoaks.com to my cloud server. Life is good. 🙂

  • Updating an old platform, and running PHP4 and PHP5 together.

    Jane and I wanted to set up WordPress blogs with all the bells and whistles, and we had a quiet “pajama” weekend on tap.  My poor old server is running an old SuSE distribution (9.2) which hasn’t been supported with updates for several years now.

    So you know what the food chain is like for manual updates. This requires that library, this patch, etc.

    My system was way behind for Apache 2, running MySQL 4.0, and PHP 4. WordPress runs fine on PHP 4 (as best I could tell), but right away we found that we wanted plugins that required PHP 5.

    I’ve come to hate distribution RPM files, and trying to navigate the weird package layouts of config files, libraries, binaries, data, and log files. I’ll venture that the problem isn’t really RPM files inherently, but the particular choices made by SuSE in (for instance) their apache distribution are just mind-blowingly convoluted. I also struggle with rpm’s myriad options and puzzling behaviors and interactions.

    So, indulge me, I decided to build everything from source and install to /usr/local (though MySQL was very good about providing pre-built binaries in non-RPM form).

    The PHP 5 build wanted Apache libraries, and rather than install an apache-dev package, I decided to build the latest from source. Apache is a nice smooth build. Their default install in /usr/local is really easy to manage (which made SuSE’s Apache choices that much more befuddling). I kept my old httpd config file and directory structure, and eventually threw out SuSE’s /etc/init.d/apache2 script in favor of a simple one that runs apachectl.

    MySQL was a much more difficult upgrade. I started off blundering right into the latest 5.1 to see what would happen. First problem, their binary wanted the latest C++ standard libraries. No one offers that as a simple binary download! (Why bother? Everyone gets it from their distribution.) That meant building gcc.

    The gnu C compiler has a few straightforward library prerequisites that I was able to build without too much trouble, but the compiler build itself is a monster, about three hours of build time. The build worked great, and I confess that I didn’t put together the required infrastructure to run a full gcc test suite — that would have been a few more hours, I’m sure.

    But there it was — a fresh new gcc and libstdc++. To get the libraries out of your build directory, you have to find them in the tree and “make install” each component individually. Not too difficult, but it’d be a nice thing to find in the documentation. Now be sure /usr/local/lib is in your load path or LD_LIBRARY_PATH includes it.

    MySQL 5.1 came up fine, which gave me the surprise discovery that I can’t upgrade directly from 4.0 to 5.1. I had to backup and install 4.1, do its upgrade, then 5.0, do its upgrade, then 5.1. In the end, all of the data survived just fine, including all the users and permissions through several admin schema updates.

    With Apache and MySQL all current and running, that left PHP5. It built and installed as an Apache module just fine. The next discovery is that at least one of the sites I host was not capable of running PHP5.

    A search found several blog entries talking about how to make PHP4 and PHP5 co-exist peacefully — run one as an Apache module, and the other as a CGI. Individual directories can be directed to the appropriate handler.

    I had no trouble getting either PHP4 or PHP5 to run as an Apache module, but it took some study to figure out why PHP wouldn’t run as a CGI using the instructions written maybe last year.

    PHP can build a command line module two ways, as a straight command line interpreter (CLI) or as a CGI processor. Granted that the CGI module has security considerations, but still many distributions provide both versions as php and php-cgi executables. There are configure options to enable or disable both.

    What they don’t tell you in the INSTALL file is that the makefile will refuse to build the CGI version if you include the apache extension system (apxs).

    So based on this experience, I have a few recommendations:

    • Don’t be shy about building from source, and installing to /usr/local. It’s nice and clean most of the time. Be sure you have /usr/local/lib in your LD_LIBRARY_PATH environment.
    • For the two versions of PHP, use a –prefix variable of /usr/local/php4 and /usr/local/php5 to keep them separate.
    • Watch out for this “feature” of PHP’s “make install”: It will put a LoadModule line in your httpd.conf file. If you end up with both PHP4 and PHP5 modules loaded, apache will not run. (Sorry for the downtime, guys!)
    • In your .htaccess files AddHandler directive, watch out for using just “php-script” as the handler type. If your httpd.conf is as muddled as mine, it’s likely to be in use somewhere else and cause confusion. I used “php5-script” and everything worked.
    • “Premature end of script headers” is a catch-all apache error that just means something went wrong with your CGI script. It’s horribly unhelpful, and could point to any number of different problems. In my case, the really BIG problem is that I was trying to do CGI scripting with the CLI version of PHP. The CLI version doesn’t bother writing headers at all.

    Maybe soon now I’ll go through all my Apache httpd.conf includes and clean everything up, then flip over PHP4 to CGI and PHP5 to apache module.

    This project made me a lot more confident about an upcoming new server build sometime later this year. That’s definitely due. And not SuSE this time, probably Ubuntu’s server edition.

    Catch you next time …