seeparsan.net

Switching your Magento development environment from Apache to nginx

Talesh — Sat, 05 Mar 2011 21:01:38 +0000

While at the Magento Imagine conference a couple weeks ago I learned that the entire Magento site and its demo sites all run on nginx instead of Apache. I was aware of that several alternatives to Apache existed and served various niches especially when it comes to delivering static HTML files at speed, however never really investigated them properly. Zeus and lighttpd were always just a blur on the horizon.

Seeing that I was consuming lots of information about node.js in the last couple weeks, I unavoidable came across infamous claim that node.js was faster than nginx by Ryan Dahl the author of node.js. Seeing that I was completely out of the loop as to how important this is, I investigated nginx a little more and realized that it was indeed a formidable server after PHP-FPM and APC were installed. I took a couple weeks, running tests off and on trying to see how much of an advantage there was to using nginx, PHP_FPM and APC. Given that now I have an awesome fast dev rig, the speed for the client showed little change in my tests even running something like Magento. But I found a difference when I started putting the servers on virtual machines via Virtual Box with limited specs (as close as possible to the t1.mirco AMI Amazon has so kindly offered me for one year) Wanting to play with new technologies, I stopped after three tests and decided that I was on board with nginx, especially since it was adopted by WordPress and Magento.

I intend to do some more rigorous testing and security analysis before launching a production site on this combo because making the move from Apache is a huge step. But first steps were getting my local versions of Magento Community Edition 1.4.2.0 and 1.5.0.1 running on nginx. Yes I know I mentioned earlier that nginx made little difference when running on a brand new dev rig so why am I doing this? I firmly believe in building your software against as close as possible technology stack as your final deployment. When I worked as the Solution Architect on the Paramount Pictures/Seagate online store, all the offshore developers were building Magento stores on Windows. We had nightmares when it came to weekly deploys yet I couldn’t change the policy of an organization half way around the world. So just to be safe, if I am going to deploy onto nginx, I am going to code on an nginx server.

I usually have multiple virtual hosts running on my dev rig to manage development of extensions against multiple versions of Magento. There is also a custom build and deployment system in the back there holding everything together with git, but that is another post. The instructions below are only really intended to move a Ubuntu 10.10 development machine to nginx away from Apache. I would advice against blindly copy-pasting what you see here into your production machine. It will probably work but I am not responsible if anything breaks.

First step is to shutdown Apache

sudo service apache2 stop

I usually try to remove all the unused packages from my system if I am not going to use them. That is why my build/deploy system is built on shell scripts as opposed to anything fancy like Capistrano, Maven etc. So the next step is removing Apache from your system. I resorted to Synaptic package manager for this because a simple sudo apt-get remove apache2 didn’t remove all the packages. Here is a list of all packages I ended up removing.

sudo aptitude remove apache2 apache2.2-common apache2.2-bin apache2-utils apache2-mpm-prefork libapache-mod-php5

At this point Synaptic package manager insisted that I install the php5-cgi package so I just went along with it assuming that I will be able to remove it later on, but also aware that PHP-FPM works with PHP in a CGI context so I left it alone.

That should take care of Apache. The next step is to install nginx and start it up

sudo aptitude install nginx

sudo service nginx start

At this point nginx should have gone to your /var/www folder and carefully dropped in a folder called nginx-default in there to not disturb your existing default folder if there is one. Absentmindedly it then forgets where it put its own docroot and points is default docroot to /var/www so when you hit your localhost in a browser, you see the default nginx 404 error page. You can drop in an index.html into /var/www to see what nginx will do.

The next step is install PHP-FPM which is a bunch of binaries that attach to PHP and run a service on your localhost on port 9000. Basically nginx queries this service for the PHP compiled data to send to the browser.

sudo aptitude install php5-fpm

sudo service php5-fpm start

If you are getting “[WARNING] [pool www] pm.start_servers is not set” and you are OCD you can set the value in /etc/php/fpm/pool.d/www.conf on line 73.

APC is a PHP opcode cache that can cache the interpreted PHP script and even variables. From my testing it provided almost negligible change on a dev machine but it may run on a public server so again I installed it here. Should there be cache refresh issues we will catch it in development as opposed to launch. Zend has a good starter article on APC if you want to learn more.

sudo aptitude install php-apc

Php should be up and running but we haven’t told nginx about it yet. There are configurations for doing so which you can get from this How To Forge tutorial, but we are not going down that path today. We are going to set up two virtual hosts which point to two installs of Magento CE 1.4.2.0 and 1.5.0.1. So I am going to make two new files in /etc/nginx/sites-available/ , one called local1420 and the other called local1501 . Here I switch over to the wisdom of the Magento Wiki page on nginx with a configuration for the nginx.conf and the individual site configurations.

I have found that the nginx.conf setting works fine, but I had to tweak the individual site configs to work properly. When I have confirmed with someone smarter than me that I am doing it as right as possible I will probably modify the Magento Wiki to reflect the changes I made. For now I will post my file here so you can compare.

The major changes I made was a result of wanting to avoid http://www.local1420.com and just sticking with http://local1420.com so I removed lines 4–10 of the Magento suggested configurations. Beyond that I was not using HTTPS on my local dev machine, so I commented out fastcgi_param HTTPS $fastcgi_https;. Also seeing that I was not developing against a multistore environment, I commented out fastcgi_param MAGE_RUN_CODE default; and fastcgi_param MAGE_RUN_TYPE store;. The last change I made was to include /etc/nginx/drop.conf which I will describe at the end of the article.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

server {
listen 80;
server_name local1501.com;
root /var/www/local1501.com;

location / {
index index.html index.php; ## Allow a static html file to be shown first
try_files $uri $uri/ @handler; ## If missing pass the URI to Magento’s front handler
expires 30d; ## Assume all files are cachable
}

## These locations would be hidden by .htaccess normally
location /app/ { deny all; }
location /includes/ { deny all; }
location /lib/ { deny all; }
location /media/downloadable/ { deny all; }
location /pkginfo/ { deny all; }
location /report/config.xml { deny all; }
location /var/ { deny all; }

location /var/export/ { ## Allow admins only to view export folder
auth_basic “Restricted”; ## Message shown in login window
auth_basic_user_file htpasswd; ## See /etc/nginx/htpassword
autoindex on;
}

location /. { ## Disable .htaccess and other hidden files
return 404;
}

location @handler { ## Magento uses a common front handler
rewrite / /index.php;
}

location ~ .php/ { ## Forward paths like /js/index.php/x.js to relevant handler
rewrite ^(.*.php)/ $1 last;
}

location ~ .php$ { ## Execute PHP scripts
if (!-e $request_filename) { rewrite / /index.php last; } ## Catch 404s that try_files miss
expires off; ## Do not cache dynamic content

fastcgi_pass 127.0.0.1:9000;
# fastcgi_param HTTPS $fastcgi_https;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
# fastcgi_param MAGE_RUN_CODE default; ## Store code is defined in administration > Configuration > Manage Stores
# fastcgi_param MAGE_RUN_TYPE store;
include fastcgi_params; ## See /etc/nginx/fastcgi_params
}
include drop.conf;
}

The last bit of configuration I added was advised by Karl Bessing Basically it involves creating a short file in /etc/nginx/ called drop.conf with the following code and including it in the server section of your virtual hosts. You can read more about what it does on his site.

1
2
3

location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }
location ~ /\. { deny all; access_log off; log_not_found off; }

And with that you should have all your configuration files all ready. Next step is to link in your available sites to your enabled sites. Usually you would use a2ensite for something like this in Apache, but I was unaware if anything similar existed for nginx, so I just linked them in manually.

sudo ln –s /etc/nginx/sites-available/local1420 /etc/nginx/sites-enabled/local1420

sudo ln –s /etc/nginx/sites-available/local1501 /etc/nginx/sites-enabled/local1501

Restart nginx and php5-fpm and you are done.

sudo service nginx restart

sudo service php5-fpm restart

If you are installing Magento from scratch you can check the box displayed during setup labelled “Web server Apache rewrites”, because it works just fine. If you have any questions or improvements, feel free to leave a comment.

Javascript Master Class

Talesh — Sat, 05 Mar 2011 02:59:57 +0000

I have read Javascript: The Good Parts* cover to cover multiple times. ( Well OK lately, its been BOF to EOF on my iPad ) When someone is as resolute and pragmatic as Douglas Crockford is to a technology as widely misunderstood and reviled as Javascript, human beings take notice. I believe it is inherent in our evolution to take notice at those who are that tenacious without being bullheaded.

Since I was learning node.js during my two month work hiatus, I thought it would be a good time to brush up on good Javascript style and eventually I started poking around his book once more, and lo, I found out that Safari Books Online** had the entirety of his Javascript Master Class available for streaming. Starting out a 5+ hour tutorial at almost midnight would not count as one of the smartest things I have done in my life, but it was worth it. Unfortunately I had just gotten past my second wind when he hit Pseudoclassical Inheritance in Javascript. By far that is one of the most convoluted processes I have ever had the pleasure to try grokking at 2am. I have had nightmares for weeks because I called it quits for the night at that point.

In all honesty, the videos are phenomenal. If you are even considering being part of the future of web application development I suggest you look at the videos. Think of it like this, if someone can preface their highly detailed and technical talk about Javascript by starting with the technology of puchcards, then that person is an Alpha in his field and you should listen to him when he speaks. I was completely taken in from the first video. Reading his book on the subject matter and listening to him speak about it are two completely different things. Also I believe he goes into a little more detail in the videos.

Another Douglas, Douglas Engelbart

Many parts of the videos I have returned to, and reviewed to make sure I absorbed all the details, but that one section about Pseudoclassical Inheritance I recoil in horror everytime I think about trying to understand it. I still have a bunch of notes of things to look into, they are mostly little sparks that lit up in my head while I was learning things about Javascript I still need to try. (One of which, was “The mother of all demos” by Douglas Engelbart which I am embarrassed to admit I have not looked at yet)

Having knowledge of the history of technology I believe is a very healthy practice. Especially with the headlong rush we technologists have found ourselves in, there is always something stirring in me to step back and look at the big picture. I believe that by studying where we have tread to get where we are technologically gives you a better understanding and appreciation for the shortcoming and awesomeness of technology today. But I think critically it gives you an opportunity to cut through the stampede of developers and find new ways to direct technology in the future. That I believe is one of Mr. Crockford’s strengths.

*Disclaimer: The Amazon referral code (if there is one here) is Mr. Crockfords. I make nothing if you click on that link.
** I am still angry at Safari Books Online after their stupid iPad app fiasco, so no link for them. Half-assed software development is a cardinal sin in my eyes. I will rant about this later.

Winter downtime

Talesh — Sat, 05 Mar 2011 00:42:47 +0000

Part of being a full time software consultant is dealing with the downtime. When you specialize in such a specific niche as I do, you are bound to have spells where there is nothing really to do except market yourself. Despite this being a full time consultant gives you the power to approach what Nigel Marsh was seeking in his Work — Life balance TED talk. Luckily for the last few years my downtimes have been aligning pretty closely to the worst winter months here (January and February) so I have been spared commuting during blizzards for two years in a row. I ramp up on a pretty exciting project on Monday so I guess today is officially the end of my winter downtime.

The key I have found to surviving downtime is “catching up”. There are millions of things that one needs to catch up on especially when it comes to technology. There is the boring catching up, like finally setting up my blog and using this domain for something more than a MX record, but still its an achievement to finally have something here again after all these years. But the fun parts of catching up are awesome. Finally my personal Wiki is now up to date as my development toolkit. All the tools, wisdom and information I gathered in working on two huge e-commerce sites in 2010 (virginmobile.ca and west49.com) are all documented and safe from my unreliable memory.

But the real joy comes in getting an opportunity for unbridled experimentation with new technologies. Its the joy of making toys. In this two weeks, I have finally had a chance to get node.js running and start learning all that is it about. For some reason server side Javascript is very exciting to me. Javascript is as amazing as it is flawed. Given that I stuck with PHP all through the Python and Ruby waves in the last decade or so, I haven’t ever had a chance to experiment with the new server technology on the scene. This time it is different. Now I am already very familiar with the language itself and already love it. I just have to see what I can do with it on the server side. Tons of little ideas are brewing in my head.

What I haven’t gotten to in the last two months includes starting to play with Arduino. I really want to start tinkering with Arduino in my spare time. The simple bridge between the virtual world of software and the physical world of matter is very very tempting. Also next on the list of things to look into is Erlang, but less so. A language that has such robustness built into is very intriguing. But it will probably have to wait until I am done experimenting with node.js and CouchDB.

All in all a very productive winter downtime.

My (Dramatized) return to Linux

Talesh — Wed, 02 Mar 2011 04:57:32 +0000

5000 blazing MIPS

*nix has always been my operating system of choice. It really was what I started computing on at the University of Waterloo and consequently has left a lasting impression on me. I am not sure if they still do it now, but I know when I started at Computer Science one of the myriad information sheets you get as a frosh in the Department of Mathematics was a list of commands for editing in vi. Those sheets even had the splotches on them that early ink based duplicating machines left. Sometimes I wonder how many years back those sheets date.

I remember the 100 level courses were all done on Windows computers with Java, but what was an eager bright eyed young Trinidadian to do in a new country and in a new school but sit down at a UNIX terminal and try to figure it out? It has been love since then. The first computer I ever purchased was an SGI Indy R5000 which I got on a insane deal used from the school newspaper where I worked. Early on I realized that if I wanted to do the “other-computer-things” that didn’t involve programming but were necessary to interacting with other people, I would need an operating system that was a little more compatible with Windows. So I bide my time until “BSD” came to Apple hardware (yup we called Mac OS X a BSD clone back then), and after working three jobs one summer bought myself a shiny little iBook (which still runs to this day).

Nostalgia and reliability aside, I only actually came to Apple for the modern OS that ran on a UNIX. I always ran my laptop alongside a UNIX machine, and eventually alongside a Linux machine. By and by eventually I started dabbling with digital media, and eventually print publishing of all things for a software developer to dabble in (remember I worked at the school newspaper). This capitulated my move completely to Mac OS X, a move that I eventually became comfortable with, despite its initial unsettling feelings. In the intervening 5–6 years, I did dark things. Primary among them was compromising my own core beliefs by pirating commercial software* to run on my Macs.

But eventually it became too much when I began seeing the piracy all around me capitulate to coworkers stealing very cheap designs from the internet to print. It reflected my own dirty perception of my own software piracy and I broke. Long grave emails were written about piracy and summarily ignored. I did some serious soul searching on the subject and realized I was wrong to go against my very own core beliefs and an the internal rebellion started. Epic instrumental music may have even been playing in the background while I did said soul searching. I am not really sure, it is all a montage to me.

I usually hold on to my hardware for a long time and push them to the limits of their functionality, but that didn’t stop me from beginning the holy purge with a complete drive reformat. I was moving to a point where I needed Creative Suite less and less, but nevertheless, I purchased it along with Microsoft Office and the latest Mac OS X. Even then I was slowly phasing out print publishing but never lost touch with Linux as I was consistently deploying server software onto it.

My mobile solution will probably always be Apple hardware I believe; mainly now just due to the fact that it has become so mainstream that I should have no problems working on location. But last year when I decided to use Ubuntu as my development rig on location building the West 49 website I was blown away. The Linux desktop experience had evolved into something modern in the last size years. Even more functional than Mac OS X in some respects. But there was more to it.

Not only was I writing software with joy, I was handily managing the roles of Software Architect, Project Manager, Developer and even Account Exec to some degree. Hammering away at a keyboard writing software on Ubuntu reinstilled a fire in me that I had long forgotten. Yes Mac OS X has a terminal, but it just wasn’t the same. Linux felt much purer and my Magento extensions didn’t have problems with non-case sensitive file systems!

My aging wheezy iMac G5 began to fail, so it was time to return to Linux. The guys at Canada Computers were well educated as usual and walked me through the ropes of what was new in consumer hardware these days and what worked well with Ubuntu. Within a couple hours I was at home looking at almost a dozen boxes of computer parts ready to jump in. No anti-static wristband needed, I was going to ride this on the wild side ready to tear in. Two mismatched parts, a rats nest of wires, three lost screws and a live USB drive later I was ready to return to my Linux.

Things that took me completely by surprise in Maverick Meerkat:

The new Ubuntu font is absolutely gorgeous (The print admirer in me may never die in this cold cold digital age)
HDMI audio and video out to my monitors just work
Multimonitor video just works
Logitech Optical Trackman works better than Mac OS X ( still cannot recognize one button though )
It was half the price of a new Mac but thats nothing new
Dropbox just works
Sadly my muscle memory on ALT and CTRL keys are Mac style now, but that key mod map change just works
Visuals are so much more refined, did I mention the drop dead gorgeous new Ubuntu font with 10.10?
There is so much more built in than Mac OS X its astounding
Connecting to a network share via AFP was more reliable on Ubuntu than on Mac OS X
Software installation has had major improvements, Ubuntu Software Centre is the open-source godfather of the Mac App Store

Nevertheless I still have tons to learn again. I used to have a setting on Mac OS X to open three terminals side by side to completely fill my screen depending on which screen I had hooked up to it. But now I want to investigate this thing I have been reading about called tmux. It would be a distinct improvement if I could just open open terminal, make it full screen and have three “terminals” side by side there to code. Thats probably a task for tomorrow to install tmux and test it while I set up my git and build systems. Blogs shall follow on my experiences with tmux.

All in all I am thrilled and comforted that I have returned back to where I was.

*Disclaimer: I now own a fully licensed version of Creative Suite and Microsoft Office, please forgive me Adobe, Microsoft. I was young and stupid.

**If you clicked on this post looking for cute furry animals, I apologize.