The previous article in this series dealt with the various external factors that impact the performance of Drupal sites, exploring ways to mitigate them. Now, let’s look at how it’s possible to increase performance without additional hardware, but by tuning the various components of the hosting stack.
Early optimisation is bad and ought to be avoided. You should embark on an optimisation exercise only when you encounter issues like degradation in response times. First analyse the reasons for the slow response of each component of the technology stack before attempting a tuning exercise. But before going further, let’s assume that your site is hosted on the LAMP stack, i.e., Linux (Centos or Ubuntu), Apache2, MySQL and PHP with Drupal 7.x; and that you have your site hosted on a VPS like Rackspace, Instacompute or on a dedicated server but not on a shared hosting environment. Let’s also assume that you are able to log in to the VPS using SSH to gain access to the shell.
Understanding your stack’s performance bottlenecks
Studies by SourceLabs have revealed that Apache is bandwidth limited, PHP is CPU limited and MySQL is memory limited. In our exercise, let’s ensure that these components use what’s available effectively rather than throwing in more of these resources.
Tools for measuring performance and monitoring
Firebug and Yslow
The Yslow add-on for Firebug (on Firefox, Chrome users have to use Firebug Light) is provided courtesy of the Yahoo Developer Network and provides a performance grade for sites along with an overall score.
Pingdom and Site24x7
The simplest way to measure performance is to use the several online tools available from Pingdom and Zoho. Pingdom is similar to Yslow; however, unlike the latter, which only tests performance from your browser, Pingdom allows testing of the site from several locations. This gives you an insight into the loading times in a particular region. In the rest of this article, I will use Pingdom to measure response times. Site24x7 is a similar offering from Zoho.
Munin
Munin is an invaluable network resource monitoring tool. A default installation provides information on server health in graphs. You can monitor the performance of almost any resource like the disk, memory, etc, on your network using Munin. You can view the health of your servers at http://munin.sastratechnologies.net
Figure 1 shows a graph of the memory usage of one the servers. The red area indicates that the server is swapping heavily. This server is badly in need of more memory and needs to be resized.
Devel()
This Drupal module provides valuable information to Drupal module developers. It logs query times and highlights slow queries. It also provides the memory consumption information when it initialises and at shutdown.
XHPRof
Originally developed by Facebook, XHPRof is a function level hierarchical profiler for PHP. This PECL package can be integrated with your Drupal installation to report CPU times and memory usage of your modules.
Command line
Some knowledge of the command line is essential to analyse the consumption of hardware resources. Though Munin plots a neat graph, the command line is essential to get the numbers.
Lets get started now. First identify the bottlenecks using Pingdom (you could use Site24x7 or Yslow, if you wish) and identify which of the components need to be fixed. Then either remove the bottleneck or adjust the parameters to tune the component.
Check your site using Pingdom
Measuring performance and tweaking your stack is iterative and you’ll usually end up doing three to five round trips. Start the exercise by checking the site using Pingdom. Visit http://tools.pingdom.com and provide your site’s URL. Within a few minutes, you’ll have a neat bar graph of all the requests done to load your URL, a performance grade score, the number of requests, the load time and the page size. In our case, mileage varied from site to site. In one case the browser waited for 5.95 seconds for the home page to load, and the total time to load the site was 11.76 seconds! In another case, the browser took 657 milliseconds to download menu.css which was about 2.4 KB. In both these cases, the root causes were more or less similar.
- There were query strings in the URL for a few images
- Two requests, one for a CSS file and another an image, were returning 404/410 responses
- JavaScript assets were not being cached
At this point you should understand that not all of these symptoms were present in a single sitethese are two different sites. However, for the rest of the article, I will refer to a single site and demonstrate how each task has improved the performance. At the point at which the sites performance is acceptable, I will illustrate further tuning exercises by picking up some of my other sites.
HTTP compression
Enabling compression on Apache decreases the response payload size. Present day browsers accept compressed content and requesting for it saves bandwidth. To enable delivery of such content, enable the mod_deflate module.
To check whether your site is actually returning content that is compressed, you can view the response headers. To do so, use the command line and type:
wget -S www.yourdomain.in
After you make these changes to Apache, test the site using Pingdom to see if there is any improvement. In our case, Pingdom returned a response time of 7.93 seconds, a significant improvement from the 11.76 seconds.
Aggregate CSS and JS assets
Yahoo recommends aggregation of CSS and JS. In Drupal, you can do this in the Home | Administration | Configuration | Development page. Check the boxes in the Bandwidth Optimization box. Pingdom reported 6.11 seconds after we enabled aggregation.
Configure a file-based caching system called Boost
Boost is a contributed Drupal module that provides static page caching. If your site receives mostly anonymous traffic you should install Boost. Before you install it, you should configure Drupal to output clean URLs for Boost to work. Clean URLs in Drupal can be enabled only if Apache has the Mod_Rewrite module enabled.
Boost generates .htaccess rules that you should copy and paste in your .htaccess folder. Remember that if you have followed the instructions in previous sections and set AllowOverride None you should enable Override for the folder in which your .htaccess resides. Boost adds its signature to each page that Drupal generates. You can view the signature by looking at the source of the page and at the very bottom you will see the following line:
<!-- Page cached by Boost @ 2013-03-30 05:51:41, expires @ 2013-04-06 05:51:41, lifetime 1 week -->
In our case, after installing Boost, Pingdom reported 4.86 seconds. This is because Boost caches the content on the disk and Drupal doesn’t have to generate the pages every time a request is received.
If your Drupal site has modules that set cookies or sessions like QuickTabs, Recaptcha, SimpleNews, Webforms, etc, you should remember that they disable caching. So its a trade-off between whether you want to retain such modules or opt for speed. You might want to look for alternatives to these modules or, better still, roll out your own module.
Using Boost does have its side effects. If you set the cache expiry to say, four weeks, and you make changes to your site, these will not be visible to the visitor till such time the new page is generated. You could schedule the Boost Crawler to generate pages at pre-determined intervals if your pages are undergoing changes at a frequency that doesn’t warrant disabling Boost. On our sites, we have disabled the crawler because it had its overheads; instead, we make it a point to delete the Boost cached pages every time we make changes.
Remove references to missing assets
The 404/410 responses that Pingdom listed were for images that were missing. We removed these references in our theme template files and Pingdom reported healthy response times.
From Amsterdam, Pingdom reported a load time of 2.27 seconds, and 1.26 seconds from Dallas. We achieved the acceptable performance figures and hence stopped further tweaking and tuning. However, for two other sites where we have over 800 unique visitors and at least a 100 logged in sessions every day, we had to implement other measures. They have been listed in the order that they were implemented.
Increase PHP memory limit
One of the methods to improve PHP performance is to increase the memory available to it by specifying a directive in php.ini. On Centos machines the file is located at /etc/php.ini while on Ubuntu it is found at /etc/php5/apache2/. Add the following line:
memory_limit = 64M ; Maximum amount of memory a script may consume (64MB) To specify unlimited memory, use a value of '-1'.
If you are unable to access the php.ini, you can specify the following directive in the .htaccess:
php_value memory_limit 64M
Alternatively, you can also specify the following directive in your settings.php file. This will affect only the site that uses this file:
ini_set('memory_limit', '64M');
Implement a PHP op-code caching mechanism called Alternative PHP Cache (APC).
All our Drupal servers have APC installed. APC is an op-code cache. It caches compiled PHP code and hence improves response times. To get your Drupal installation to use APC, you will have to install the APC binaries either using your Linux distribution’s package manager or using PECL. After installing it, add the following line to /etc/php5/apache2/php.ini.
extension=apc.so apc.enabled=1 apc.shm_segments=1 apc.shm_size=64 apc.stat=0 apc.ttl=0
Then restart Apache and check php.ini in the Drupal status page to see if APC is installed. Check if the shared memory size for APC is sufficient for your page size. The default apc.shm_size parameter is set to 32 MB; increase it to 64 MB (you can go up to 96 MB).
Implement a memory based caching mechanism called Memcached
Memcached is a distributed object caching system. We install Memcached on all our Drupal servers. You can install the Memcached binaries using your Linux distribution’s package manager. Use PECL to install the Memcached PHP extension, and then add the following lines to /etc/php5/conf.d/memcache.ini
extension=memcache.so memcache.hash_strategy="consistent"
If you have fiddled with the iptables while setting up your VPS, modify the rules to enable port 11211the port on which Memcached listens; and in /etc/memcached.conf, look for -m 64 and change it to -m 256 to allow 256 MB RAM for Memcached.
Start the Memcached service, restart Apache and install the Drupal Memcached module to allow Drupal to use Memcached; specify the following in your settings.php file:
$conf['cache_backends'][]= 'sites/all/modules/memcache/memcache.inc'; $conf['cache_default_class'] = 'MemCacheDrupal'; $conf['memcache_key_prefix'] = 'unique_string';
Memcached will cache DB queries and provide a significant performance improvement for Drupal sites.
Configure MySQL for optimum performance
Some of our installations have a dedicated Virtual Private Server for the database. If you have a server dedicated to MySQL and it has a memory of 1 GB, you can start with the following values in your configuration file (/etc/mysql/my.cnf).
key_buffer = 256M query_cache_size = 128MB query_cache_limit = 4MB table_cache = 512 sort_buffer_size = 32M myisam_sort_buffer_size = 32M tmp_table_size = 64MB delay_key_write = 1 wait_timeout = 60
The possibilities of improving database (DB) performance are endless and there is no one magic solution. Rather than embarking on an elaborate DB tuning exercise, I believe the biggest performance improvements come from identifying and tuning your slow DB queries. Activate the slow query log and identify the slow queries and rewrite them. You should do this especially if you have installed the views module. You could also use Devel to identify the queries that take a lot of time. Having identified the slow queries, change them to make them execute faster.
Check your code, and follow the best practices in your modules
Finally, if you have written custom modules then use the Devel module to identify slow queries and rewrite them. Use the XHProf module (along with the XHProf php extension) to profile memory and CPU cycles, and refactor your code.
In the book ‘High Performance Websites’, Steve Sounders mentions some rules to speed up your website. Keep them in mind while writing your modules. Ensure that you don’t embed PHP code, CSS and JavaScript in your content pages.
The performance techniques described in this article are sufficient for most Drupal installations.
HTTP caching using Apache
Caching frequently accessed content at the browser or proxy can improve response times because the content will not be retrieved from the server. This saves bandwidth, decreases the load on the server and reduces latency. Enable the mod_expires and mod_headers modules. To enable caching at the server side, enable the mod_cache module. You could also use Google’s mod_pagespeed.
Separate servers for static and dynamic content
Apache processes serving Drupal sites are between 40 MB to 60 MB. Their size grows to accommodate the content being served and doesn’t decrease till the process dies. Let’s assume that your home page consists of several images and the browser makes multiple requests to retrieve them. Assuming that we have the KeepAlive directive set to ‘On’, each request is serviced by an Apache process that is 60 MB in size, even though the request is for an image that can be served by a 1 to 3 MB process.
You could set up a Baby Apache compiled with just the minimum required modules to serve static content like images. The Baby Apache will receive all client requests. It should handle requests for images and it should forward all other requests to the heavyweight Apache that is serving the dynamic content. You will need to enable the mod_proxy and mod_rewrite modules, and set up the rewrite rules in your configuration file.
Using a content delivery network (CDN)
We use CDNs sparingly. Consider a CDN only if your site has a reasonable number of visitors from a region. Do not include a CDN in your stack just to increase the throttling of your site; the added maintenance is not worth the effort. In the initial days when we had just two sites, we hosted them on Rackspace because it had a 256 MB offering, which we found very attractive. However, when most of our visitors (at least 99 per cent of them were from India) complained of slow response times, which was due to the network latency, we tried the CDN (Cloudflare) route as a quick fix solution and the problem got compounded. The CDN wouldn’t refresh content and visitors were being served stale pages. I suggest that, as a first step, you should move your hosting to a region where most of your traffic originates. For the others, consider implementing a reverse proxy like Squid or Varnish.
output_handler = ob_gzhandler
Use a reverse proxy
After Boost, adding a reverse proxy will give your site one of the biggest performance improvements. There are two candidates for thisSquid and Varnish. Though opinion on which one to use for Drupal sites is divided, I am in the process of configuring Varnish on one of the Rackspace servers in the UK. Varnish provides its own scripting language, VCL, and offers acceleration by a factor of 300 to 100 times.
Replace Apache with Nginx
Nginx is a high performance HTTP server. You could consider replacing Apache with Nginx. You should keep in mind that Boost has limited support for Nginx.
Specify an output handler for PHP
If you are constrained for bandwidth, specify the following
directive in php.ini:
With this, output is compressed for browsers that accept compressed content.
Optimise PHP to Apache communication
To optimise PHP to Apache communication, set the directive in php.ini by uncommenting the following lines:
; output_buffering ; Default Value: Off ; Development Value: 4096 ; Production Value: 4096
Use the PHP tidy extension
The PHP tidy extension is used to remove the white spaces in rendered HTML so that the payload sent to the browsers is smaller.
It is possible to increase the performance of a Drupal site without adding additional hardware by tuning the various components of the hosting stack, especially by configuring various caching mechanisms .
References
[1] CERT 7 Test Results http://web.archive.org/web/20061115173834/http://sourcelabs.com/pdfs/SourceLabsApacheMySQLPHPTestResults.pdf
[2] http://tools.pingdom.com/fpt/ and https://www.site24x7.com/
[3] http://munin-monitoring.org/
[4] Httpfox (https://addons.mozilla.org/en-US/firefox/addon/httpfox/). Other tools that you could use are Httpie (https://github.com/jkbr/httpie), Fiddler (http://www.fiddler2.com/fiddler2/) or plain old Telnet.
[5] http://developer.yahoo.com/performance/rules.html
[6] Boost (http://drupal.org/project/boost)
[7] Other Options: TUX / “Red Hat Content Accelerator” – http://www.redhat.com/docs/manuals/tux/, kHTTPd – http://www.fenrus.demon.nl/, thttpd http://www.acme.com/software/thttpd/, lighttpd (), lingered (http://www.iagora.com/about/software/lingerd/)
[8] http://www.squid-cache.org/
[9] http://varnish-cache.org/
[10] thttpd – http://www.acme.com/software/thttpd/, Caudium – http://caudium.net/index.html, Roxen – http://www.roxen.com/products/webserver/, Zeus ($$) – http://www.zeus.co.uk
Nice article. Plan to test many of these recommendations on my private site running on Ubuntu. Thanks for sharing.