Morris "Mojo" Jones

Mojo's blog banner

Tag: linux

  • Ruby gem problems around libv8, therubyracer, mac vs. linux, native extensions

    I haven’t done a techie blog post for a while, and this “solved problem” keeps raising its head at work, so here goes.

    Here’s the situation:

    • A ruby web app
    • Develop on Mac OS X
    • Deploy on Linux
    • Using bundler to control gem versions
    • Using therubyracer and libv8

    You build your app on your Mac, install your bundle of gems using bundler, and your tests run fine. Now you deploy the app to your production system, and there’s this error:

    Some gems seem to be missing from your vendor/cache directory.
Could not find libv8-3.11.8.13 in any of the sources

    Or worse, this error:

    Installing therubyracer (0.11.3) with native extensions 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

        /usr/local/bin/ruby extconf.rb --with-ruby-include=/usr/src/ruby-1.9.3-p392
checking for main() in -lpthread... yes
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

[...]
/usr/local/lib/ruby/gems/1.9.1/gems/libv8-3.11.8.13-x86_64-linux/ext/libv8/location.rb:15:in `initialize': 
Permission denied - /usr/local/lib/ruby/gems/1.9.1/gems/libv8-3.11.8.13-x86_64-linux/ext/libv8/.location.yml (Errno::EACCES)

    If you’re careful with your production deploys, then

    • Your app does not run as root on the production servers, and
    • Your app’s gems are cached in vendor/cache, thus not pulling gems from rubygems.org, or anywhere else.

    Here’s the thing with libv8: it’s designed to be optimized for each platform where it runs, so the gem has native extensions with different versions for Mac OS X and Linux. Your gem cache needs to have both native versions of the gem in vendor/cache to satisfy the two platforms. When you bundle the gems on your development machine it doesn’t build the linux native version (obviously).

    When you deploy the app on Linux, bundler fails because it doesn’t find the Linux custom gem.

    Under the deployment conditions I described, trying to build libv8 fails trying to write to the system gems, hence the permission error I mentioned. (I have no idea what therubyracer is doing trying to write into the libv8 gem!)

    So here’s how to solve the problem.

    First, on your development system, do the normal bundle install, verify that the darwin (Mac OS X) version of libv8 is found in vendor/cache:

    $ ls -l libv8*
-rw-r--r-- 1 mjones mjones 33652224 2013-02-26 10:16 libv8-3.11.8.13-x86_64-darwin-11.gem

    Now commit your changes with the new gem file, and push it to your git repository.

    Next, connect to a handy Linux server. Install rvm (or rbenv) and your current Ruby, with a nice clean gemset. Be sure you have bundler installed.

    (You may need some bundle options):

    bundle config build.linecache19 --with-ruby-src=/usr/src/ruby-1.9.3p392
etc.

    Clone your git repo, and do a bundle install. Using rvm (or rbenv) means you’ll have local copies of ruby and your gem set, and that should avoid any permissions problems.

    Now check vendor/cache, and you should have a Linux version of the libv8 gem available. To deploy on both platforms, keep both in vendor/cache. Bundler has been good about keeping both versions around and not trying to clean the one you’re not using:

    $ ls -l libv8*
-rw-r--r-- 1 mjones mjones 33652224 2013-02-26 10:16 libv8-3.11.8.13-x86_64-darwin-11.gem
-rw-r--r-- 1 mjones mjones  3144704 2013-02-26 11:29 libv8-3.11.8.13-x86_64-linux.gem

    For good measure, I also often include the generic version of the gem, downloaded manually from rubygems. Add your updates to your git repository and push them to the remote origin.

    You should find that your app deploys successfully on Linux now.

  • Perfect wallpaper from digital photos using Linux and Netpbm

    The world is full of wallpaper managers for every operating system out there. I enjoy wallpapers taken from some of my digital photography, such as this trip to Yosemite last year.

    On nice modern monitors, you can really enjoy the full resolution of your pictures. Jane and I just replaced our old Viewsonic CRT monitors with some nice Dell 23-inch LCD models.

    Immediately I saw that I needed to regenerate our collection of wallpaper photos to match the aspect ratio and higher resolution of our new monitors. I admit to being a stickler for my wallpaper photo albums. I have these requirements:

    • I want to scale and crop the photos to fill the screen exactly, no tiling or stretching.
    • I want no black bars, letterboxing, or distorted aspect ratios
    • I want to dim the maximum brightness of the photos so my desktop icons are still discernable

    My Canon 20D full-resolution pictures have more than enough pixels to fill the biggest screen, so I cobbled together a shell script some time ago using the PBM (Portable Bitmap) tools that have been around since probably the 80’s for manipulating images. I don’t think many of the Linux distros install the toolset by default, but they’re easily available. On Ubuntu or Debian you can install them with “apt-get install netpbm”.

    I start by making a work directory (“wallpaper” in this instance) and a subdirectory to hold the full-resolution original images, named “full”. I collect copies of my full-resolution pictures there in ~/wallpaper/full.

    Next I need to work out the transform. My original resolution images are 3504 pixels wide by 2336 pixels vertically. My monitor is 2048 pixels by 1152 pixels.

    Rather than work out the math, I just scaled an original picture to the monitor width to see how tall it would be. This command pipeline would scale a picture to 2048 pixels wide:

    jpegtopnm full/IMG_1234.jpg | pnmscale -width 2048 | pnmtojpeg >IMG_1234.jpg

    Opening that scaled image in Gimp told me that it was 2048 x 1365. That tells me that I need to crop some lines from the top and bottom of the image to fit them exactly to my monitor field. 1365 – 1152 leaves 213 lines to cut from the image. With the pbmtool “pamcut” I plan to cut 107 lines from the top of the image and give it a total height of 1152.

    So I made this shell script to process all of the photos. The plan is to read all of the files from the “full” directory, and write perfectly scaled images to a subdirectory named “2048”. I’m also going to use the “ppmdim” utility to reduce the overall brightness of the images just a little. Here is the final script, called “mkwall2048”:

    for i in `ls -1 full`
do
echo $i
jpegtopnm full/$i \
 | pnmscale -width 2048 \
 | pamcut -top=107 -height=1152 \
 | ppmdim 0.8 \
 | pnmtojpeg >2048/$i
done

    This loops through every file in the “full” directory, putting the filename in variable $i. The rest of the script is a pipeline that feeds the image through five different tools from the Portable Bitmap collection, as follows:

    1. jpegtopnm converts the input file to a portable “any” map, then feeds it to stdout
    2. pnmscale scales the image to a width of 2048 pixels, preserving the aspect ratio
    3. pamcut slices off the top 107 lines, and preserves the next 1152 lines of the image
    4. ppmdim reduces the brightness of the image by 20% (80% of the existing brightness)
    5. pnmtojpeg converts the portable bitmap image back to a JPEG file

    I’ve adjusted this script using the same process to make perfect wallpapers for my laptop monitors and desktops at work. It’s a real treat having a slideshow of my favorite photography available behind my work.