CentOS 7.2.1511 and Asterisk 13.1: max open files problem

Asterisk 13.1 (running under CentOS 7.2.1511, maybe some other Asterisk versions too) has a problem: setting ‘maxfiles’ in asterisk.conf does not matter.
So, one day, your Asterisk may stop processing new calls under high load, and as the max opened files limit is reached, you will even have no opportunity to enter the Asterisk CLI.

The /var/log/asterisk/messages (or Asterisk CLI, if you’ve been already logged in before reaching the max open files limit) will be filling with messages like these:

[Feb 13 16:47:00] WARNING[9283] acl.c: Cannot create socket
[Feb 13 16:47:00] WARNING[9283] rtp.c: Unable to allocate RTCP socket: Too many open files
[Feb 13 16:47:00] WARNING[14732] acl.c: Cannot create socket
[Feb 13 16:47:00] WARNING[14732] channel.c: Channel allocation failed: Can't create alert pipe! 
Try increasing max file descriptors with ulimit -n

As a workaround, you may change the soft and hard limit for the Asterisk process on the OS level. But first of all let’s check how many files are allowed for Asterisk to be opened:

voip-b2b01 ~ # cat /proc/`pidof asterisk`/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             15693                15693                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       15693                15693                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

So, you see the line with ‘1024’ value, in spite of our setting in asterisk.conf:

Max open files            1024                 1024                 files     

Let’s fire! CentOS has a nice tool called prlimit. Use it:

prlimit --pid `pidof asterisk` --nofile=8192

Now let’s check again:

voip-b2b01 ~ # cat /proc/`pidof asterisk`/limits
Limit                     Soft Limit           Hard Limit           Units     
...
Max open files            8192                 8192                 files     
...

CentOS 7.2 uses Systemd, so it’s wise to edit Asterisk-related configuration file.
We check the /etc/systemd/system/multi-user.target.wants/ directory and see an ‘asterisk.service’ file there, which is a symlink in fact, leading to /usr/lib/systemd/system/asterisk.service:

voip-b2b01 ~ # ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx  1 root root 40 июл 14 13:09 asterisk.service -> /usr/lib/systemd/system/asterisk.service
lrwxrwxrwx. 1 root root 38 сен 15  2015 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root 41 июн 28 14:44 bacula-fd.service -> /usr/lib/systemd/system/bacula-fd.service
lrwxrwxrwx. 1 root root 39 сен 15  2015 chronyd.service -> /usr/lib/systemd/system/chronyd.service
lrwxrwxrwx. 1 root root 37 сен 15  2015 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 42 сен 15  2015 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 41 июн 28 12:33 nfs-client.target -> /usr/lib/systemd/system/nfs-client.target
lrwxrwxrwx. 1 root root 39 июн 28 12:49 oddjobd.service -> /usr/lib/systemd/system/oddjobd.service
lrwxrwxrwx. 1 root root 40 сен 15  2015 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 39 сен 15  2015 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx  1 root root 40 июл 14 13:02 sendmail.service -> /usr/lib/systemd/system/sendmail.service
lrwxrwxrwx  1 root root 41 июл 14 13:02 sm-client.service -> /usr/lib/systemd/system/sm-client.service
lrwxrwxrwx. 1 root root 36 сен 15  2015 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 36 июн 28 12:49 sssd.service -> /usr/lib/systemd/system/sssd.service
lrwxrwxrwx. 1 root root 44 июн 28 14:19 zabbix-agent.service -> /usr/lib/systemd/system/zabbix-agent.service

Just uncomment/edit the parameter LimitNOFILE. And I advise you to uncomment/edit the Restart=always line to let Systemd control the Asterik is running (now you need not use Monit). You will have such a configuration:

voip-b2b01 ~ # grep -vE '^$|^;|^#' /usr/lib/systemd/system/asterisk.service 
[Unit]
Description=Asterisk PBX and telephony daemon.
After=network.target

[Service]
Type=simple
Environment=HOME=/var/lib/asterisk
WorkingDirectory=/var/lib/asterisk
User=asterisk
Group=asterisk
ExecStart=/usr/sbin/asterisk -f -C /etc/asterisk/asterisk.conf
ExecStop=/usr/sbin/asterisk -rx 'core stop now'
ExecReload=/usr/sbin/asterisk -rx 'core reload'
LimitNOFILE=8192:8192
Restart=always

PrivateTmp=true

[Install]
WantedBy=multi-user.target

Don’t forget to re-read settings by systemd:

systemctl daemon-reload

… and now it’s time to restart Asterisk:

systemctl restart asterisk.service

Well, at this point we’ve changed the max open files limit for the Asterisk process manually and changed the Systemd Asterisk config, hoping that it will start now with the 8192 max open files limit.

If you’re interested in how many files are currently opened by your Asterisk, you may try:

voip-b2b01 ~ # lsof -p `pidof asterisk` | wc -l
956

Tags:

Comments are closed.