Turn off swap

swapoff -a
sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab

Raising the Maximum Number of File Descriptors (Open Files) on Ubuntu 14.04 Trusty

So, I think I reached a new milestone in my sysadmin career (I mainly identify as a developer, but am responsible for the infrastructure at Allmyles as well, currently.) The milestone I’m talking about is when you’re first hit with a downtime completely out of the blue, seeing some weird ‘Reached maximum number of open files’ error, or something similar.

Chances are, if you now need to research how to raise the limit, you didn’t even know before that such a limit existed, or like me, you had some vague idea, but never gave it much thought ('We don’t need to deal with this right now, surely we will revisit the issue before we even get close to the current limit.’) Well, no. You won’t. And when your services go down, it’s going to be hell to find any worthwhile resource that describes how to solve this thing, so you better do this now, or at the very least, set up some monitoring for the number of used file descriptors for critical processes!

If anyone’s interested, as I understand it, this entire feature is mostly obsolete — it seems like a nice safeguard against a certain type of memory leak (namely, accidentally opening too many files which each take up a bit of year memory), or for controlling resources available for each user, but it seems like it has practically no use nowadays, when file opens have practically no cost at all on most systems.

For reference, our current stack, for which the steps described below worked is:

• Instances hosted on Amazon EC2 (not that this one should matter)
• Ubuntu 14.04 (Trusty) for the OS
• Supervisor for process management
• And lastly, the process hitting errors was Elasticsearch

Realizing You Have A Problem is the First Step

If you’re currently trying to fix the issue on a live server, just skip on down to the next section. Otherwise, it’s probably a nice idea to check what your limit is set to, and how close you are to reaching it.

The canonical way to check the limits for your current session which everyone will tell you is the ulimit command:

$ ulimit -n
4096
$ ulimit -Hn
4096
$ ulimit -Sn
4096

A couple things to note:

• There are separate limits for different users, so make sure to run this as the user your process is using.
• There’s a hard limit, and a soft limit. The latter is the actual limit your processes have to obey, and the former set the maximum number the soft limit can be set to. If you need to set separate values for these two, you probably already know how to do that and are not reading this post, so just keep in mind to always modify both, and check the soft limit.

So this is what you would find after 10 seconds of Googling, but keep in mind that ulimit is not guaranteed to give you the limits your processes actually have! There’s a million things that can modify a limits of a process after (or before) you initialized your shell. So what you should do instead is fire up top, htop, ps, or whatever you want to use to get the ID of the problematic process, and do a cat /proc/{process_id}/limits:

$ cat /proc/1882/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15922                15922                processes
Max open files            4096                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15922                15922                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Eek! Our Elasticsearch process has a max file limit of 4096, which is way less than we intended! Lucky for us /proc/{process_id}/fd is a directory that holds a file for each open file the process has, so it’s pretty easy to count how close we are to reach the limit:

$ sudo ls /proc/1882/fd | wc -l
4096

Welp, at least that explains why we’re seeing all those errors in the log. For the record, it took us 79 Elasticsearch indices to hit the 4096 open file limit. Oh well, let’s move on to actually fixing this.

The Stuff You Came Here to Read: Raising the Limit

Sorry it took this long to get here! The ulimit -n 64000 command that’s floating around, as every easy 'solution’, will not actually fix your problem. The issue is that the command only raises your limit for the active shell session, so it’s not permanent, and it most definitely will not affect your processes that are already running (actually, nothing will, so don’t have high expectations here.)

The actual way to raise your descriptors consists of editing three files:

• /etc/security/limits.conf needs to have these lines in it:

  *    soft nofile 64000
  *    hard nofile 64000
  root soft nofile 64000
  root hard nofile 64000
  

  The asterisk at the beginning of the first two lines means 'apply this rule to all users except root’, and you can probably guess that the last two lines set the limit only for the root user. The number at the end is of course, the new limit you’re setting. 64000 is a pretty safe number to use
• /etc/pam.d/common-session needs to have this line in it:

  session required pam_limits.so
  

• /etc/pam.d/common-session-noninteractive also needs to have this line in it:

  session required pam_limits.so
  

I never got around to looking into what exactly this does, but I’d assume that these two files control whether the limits file you edited above is actually read at the beginning of your sessions.

So, you did it, great job! Just reboot the machine (yup, sadly, you need to) and your limits should reflect your changes:

$ ulimit -n
64000
$ ulimit -Hn
64000
$ ulimit -Sn
64000

Whee! And to check your problematic process again:

$ cat /proc/1860/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15922                15922                processes
Max open files            4096                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15922                15922                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Wait, what? That still says 4096! There must be something we missed (except if you’re seeing 64000–in which case you can pat yourself on the back and close this tab.)

Finding That One Last Thing You Still Need to Do

So you stitched together and followed all the steps that you found on three different blogs, and four Stack Overflow questions, and it still won’t work. Huh?

The thing that most resources neglect to emphasize, is that your limits can really easily be modified by anything responsible for execution of your processes. If ulimit -n (run as the correct user) is giving you the number you just set, but cat /proc/{process_id}/limits is still printing the low number, you almost certainly have a process manager, an init script, or something similar messing your limits up. You also need to keep in mind that processes inherit the limits of the parent process. This last step (if necessary) is going to vary a lot based on your stack and configuration, so the best I can do is give you an example, of how our Supervisor setup had to be configured to fix the number of file descriptors.

Supervisor has a config variable that sets the file descriptor limit of its main process. Apparently, this setting is in turn inherited by any and all processes it launches. What’s even worse is that the default for this setting is 4096, and that’s no good when we’re wanting 64000 instead. To override this default, you can add the following line to/etc/supervisor/supervisord.conf, in the [supervisord] section:

minfds=64000

Pretty easy to fix, but also really hard to find if you have no knowledge of this inheritance rule, or the fact that Supervisor by default is not having any of that OS level limit configuration. At this point, all you have to do is restart supervisord, in my case withsudo service supervisor restart. This should automatically restart your problematic processes as well, with your newly set limit:

$ cat /proc/1954/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15922                15922                processes
Max open files            64000                64000                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15922                15922                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Neat! You got that thing fixed. Now go celebrate by setting this up on your other servers, too. And while you’re at it, you can tack on some monitoring as well.

install php7 normalizer

aptitude install php7.0-intl

nginx Tuning

nginx is fast, really fast, but you can adjust a few things to make sure it's as fast as possible for your use case. Santa doesn't like it when you spend your hard earned money on extra server resources you don't really need.

The easy stuff

The easiest thing to set in your configuration is to setup the right number of workers and connections. This tells nginx to only have a single worker process. This might be appropriate for a lower traffic site, where you have nginx, database, and your app all running on the same server.

worker_processes 1;

But if you have a higher traffic site or a dedicated instance for nginx, you want this to be set to one worker per CPU core. nginx provides a nice option that tells it to automatically set this to one worker per core like this:

worker_processes auto;

nginx is much more efficient about handling connections than many other web servers, so it's usually a safe bet to set the connections to a high value. We'll also set nginx to use epoll to ensure we can handle a large number of connections optimally and direct it to accept multiple conncetions at the same time. Our config should now look like this:

worker_processes auto;
events {
    use epoll;
    worker_connections 1024;
    multi_accept on;
}

Assuming a system with 4 cores, this would allow us to have 4096 simultaneous connections. If you adjust your kernel's open file limits up you can easily bump up worker_connections higher to 2048 or 4096 allowing more connections than marketing can likely drive traffic to your site.

Since we will likely have a few static assets on the file system like logos, CSS files, Javascript, etc that are going to be commonly used across your site it's quite a bit faster to have nginx cache these for short periods of time. Adding this outside of the events block tells nginx to cache 1000 files for 30 seconds, excluding any files that haven't been accessed in 20 seconds, and only files that have 5 times or more. If you aren't deploying frequently you can safely bump up these numbers higher.

open_file_cache max=1000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 5;
open_file_cache_errors off;

Socket Stuffers

Since we're now setup to handle lots of connections, we should allow browsers to keep their connections open for awhile so they don't have to reconnect to as often. This is controlled by the keepalive_timeout setting. We're also going to turn on sendfile support, tcp_nopush, and tcp_nodelay. sendfile optimizes serving static files from the file system, like your logo. The other two optimize nginx's use of TCP for headers and small bursts of traffic for things like Socket IO or frequent REST calls back to your site.

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
}

As we mentioned in yesterday's article on Front End Performance, nearly every browser on earth supports receiving compressed content so we definitely want to turn that on. These also go in the same http section as above:

gzip on;
gzip_min_length 1000;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

Walking in a WSGI Wonderland

CI bells ring, are you listening?
In the rack, servers are glistening
We’re deploying tonight
Walking in a WSGI wonderland
What the Elves sing during deployments

Many of our readers use Python and Django, so you're very likely using nginx to reverse proxy to WSGI processes. This is pretty easy to configure, but most of the time people don't optimize this bit at all.

You'll need to adjust this for your particular site, but using something like this:

location / {
    proxy_buffers 8 24k;
    proxy_buffer_size 2k;
    proxy_pass http://127.0.0.1:8000;
}

proxy_buffering is turned on by default with nginx, so we just need to bump up the sizes of these buffers. The first directive, proxy_buffers, is telling nginx to create and use 8 24k buffers for the response from the proxy. The second directive is a special smaller buffer that will just contain the HEAD information, so it's safe to make that smaller.

So what's this do? Well when you're proxying a connection nginx is playing the middle man between the browser and your WSGI process. As the WSGI process writes data back to to nginx, nginx stores this in a buffer and writes out to the client browser when the buffer is full. If we leave these at the defaults nginx provides (8 buffers of either 4 or 8K depending on system), what ends up happening is our big 50-200K of HTML markup is spoon fed to nginx in small 4K bites and then sent out to the browser.

This is sub-optimal for most sites. What we want to have happen is for our WSGI process to finish and move on to the next request as fast as possible. To do this it needs nginx to slurp up all of the output quickly. Increasing the buffer sizes to be larger than most (or all) of the markup size of your apps pages let's this happen.

Safety

Another thing to think about is keeping people on Santa's naughty list from causing too much trouble. The way to do this is with rate limiting. While this isn't sufficient protection from something like a DDoS, it's enough to keep your site protected from smaller floods of traffic.

Rate limiting in nginx is pretty easy to setup and fairly CPU/memory efficient so good elves turn it on. Our friend Peter over at Lincoln Loop has a great write up on rate limiting with nginx to get you started.

We hope this helps eek out a bit more performance from nginx for you!

Set up a up-to-date server

curl http://nginx.org/keys/nginx_signing.key | apt-key add -
echo -e "deb http://nginx.org/packages/mainline/ubuntu/ `lsb_release -cs` nginx\ndeb-src http://nginx.org/packages/mainline/ubuntu/ `lsb_release -cs` nginx" > /etc/apt/sources.list.d/nginx.list
aptitude update
aptitude install nginx

curl https://pkg.cloudflare.com/pubkey.gpg | apt-key add -
echo -e "deb http://pkg.cloudflare.com/ `lsb_release -cs` main" > /etc/apt/sources.list.d/cloudflare-main.list
aptitude update
aptitude install railgun-stable

====================================================================
aptitude install build-essential fail2ban dstat
aptitude install python-software-properties


Install Nginx Latest Version:

add-apt-repository ppa:nginx/development
aptitude update
aptitude install nginx


Install PHP Latest Version:
aptitude install php7.0-fpm php7.0-cli php7.0-curl php7.0-mysql php7.0-dev php7.0-cli php7.0-common php7.0-mbstring

add-apt-repository ppa:ondrej/php
aptitude update
aptitude install php7.0-fpm

Install Latest GeoIP:

aptitude install libgeoip-dev
aptitude install git
git clone https://github.com/Zakay/geoip.git
cd geoip
phpize
./configure --with-php-config=/usr/bin/php-config
make
make install
echo "extension=geoip.so" > /etc/php/mods-available/geoip.ini
ln -s /etc/php/mods-available/geoip.ini /etc/php/7.0/cli/conf.d/20-geoip.ini
ln -s /etc/php/mods-available/geoip.ini /etc/php/7.0/fpm/conf.d/20-geoip.ini

Install Latest Imagick:
aptitude install libmagickwand-dev libmagickcore-dev
git clone https://github.com/mkoppanen/imagick.git
cd imagick
phpize
./configure --with-php-config=/usr/bin/php-config
make
make install
echo "extension=imagick.so" > /etc/php/mods-available/imagick.ini
ln -s /etc/php/mods-available/imagick.ini /etc/php/7.0/cli/conf.d/20-imagick.ini
ln -s /etc/php/mods-available/imagick.ini /etc/php/7.0/fpm/conf.d/20-imagick.ini


Install OpenCC:
aptitude install cmake doxygen
git clone https://github.com/BYVoid/OpenCC.git
cd OpenCC
make
sudo make install
git clone https://github.com/shtse8/opencc4php7-New
cd opencc4php7-New
phpize
./configure
make
sudo make install
echo "extension=opencc.so" > /etc/php/mods-available/opencc.ini
ln -s /etc/php/mods-available/opencc.ini /etc/php/7.0/cli/conf.d/20-opencc.ini
ln -s /etc/php/mods-available/opencc.ini /etc/php/7.0/fpm/conf.d/20-opencc.ini

Setup Web Server

Latest Nginx

curl http://nginx.org/keys/nginx_signing.key | apt-key add -
echo -e "deb http://nginx.org/packages/mainline/ubuntu/ `lsb_release -cs` nginx\ndeb-src http://nginx.org/packages/mainline/ubuntu/ `lsb_release -cs` nginx" > /etc/apt/sources.list.d/nginx.list
aptitude update
aptitude install nginx

Install FFMPEG on Ubuntu

add-apt-repository ppa:mc3man/trusty-media

aptitude update

aptitude install ffmpeg