CentOS 7.2.1511 and Asterisk 13.1: max open files problem

December 18th, 2016

Asterisk 13.1 (running under CentOS 7.2.1511, maybe some other Asterisk versions too) has a problem: setting ‘maxfiles’ in asterisk.conf does not matter.
So, one day, your Asterisk may stop processing new calls under high load, and as the max opened files limit is reached, you will even have no opportunity to enter the Asterisk CLI.

The /var/log/asterisk/messages (or Asterisk CLI, if you’ve been already logged in before reaching the max open files limit) will be filling with messages like these:

[Feb 13 16:47:00] WARNING[9283] acl.c: Cannot create socket
[Feb 13 16:47:00] WARNING[9283] rtp.c: Unable to allocate RTCP socket: Too many open files
[Feb 13 16:47:00] WARNING[14732] acl.c: Cannot create socket
[Feb 13 16:47:00] WARNING[14732] channel.c: Channel allocation failed: Can't create alert pipe! 
Try increasing max file descriptors with ulimit -n

As a workaround, you may change the soft and hard limit for the Asterisk process on the OS level. But first of all let’s check how many files are allowed for Asterisk to be opened:

voip-b2b01 ~ # cat /proc/`pidof asterisk`/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             15693                15693                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       15693                15693                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

So, you see the line with ‘1024’ value, in spite of our setting in asterisk.conf:

Max open files            1024                 1024                 files     

Let’s fire! CentOS has a nice tool called prlimit. Use it:

prlimit --pid `pidof asterisk` --nofile=8192

Now let’s check again:

voip-b2b01 ~ # cat /proc/`pidof asterisk`/limits
Limit                     Soft Limit           Hard Limit           Units     
...
Max open files            8192                 8192                 files     
...

CentOS 7.2 uses Systemd, so it’s wise to edit Asterisk-related configuration file.
We check the /etc/systemd/system/multi-user.target.wants/ directory and see an ‘asterisk.service’ file there, which is a symlink in fact, leading to /usr/lib/systemd/system/asterisk.service:

voip-b2b01 ~ # ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx  1 root root 40 июл 14 13:09 asterisk.service -> /usr/lib/systemd/system/asterisk.service
lrwxrwxrwx. 1 root root 38 сен 15  2015 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root 41 июн 28 14:44 bacula-fd.service -> /usr/lib/systemd/system/bacula-fd.service
lrwxrwxrwx. 1 root root 39 сен 15  2015 chronyd.service -> /usr/lib/systemd/system/chronyd.service
lrwxrwxrwx. 1 root root 37 сен 15  2015 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 42 сен 15  2015 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 41 июн 28 12:33 nfs-client.target -> /usr/lib/systemd/system/nfs-client.target
lrwxrwxrwx. 1 root root 39 июн 28 12:49 oddjobd.service -> /usr/lib/systemd/system/oddjobd.service
lrwxrwxrwx. 1 root root 40 сен 15  2015 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 39 сен 15  2015 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx  1 root root 40 июл 14 13:02 sendmail.service -> /usr/lib/systemd/system/sendmail.service
lrwxrwxrwx  1 root root 41 июл 14 13:02 sm-client.service -> /usr/lib/systemd/system/sm-client.service
lrwxrwxrwx. 1 root root 36 сен 15  2015 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 36 июн 28 12:49 sssd.service -> /usr/lib/systemd/system/sssd.service
lrwxrwxrwx. 1 root root 44 июн 28 14:19 zabbix-agent.service -> /usr/lib/systemd/system/zabbix-agent.service

Just uncomment/edit the parameter LimitNOFILE. And I advise you to uncomment/edit the Restart=always line to let Systemd control the Asterik is running (now you need not use Monit). You will have such a configuration:

voip-b2b01 ~ # grep -vE '^$|^;|^#' /usr/lib/systemd/system/asterisk.service 
[Unit]
Description=Asterisk PBX and telephony daemon.
After=network.target

[Service]
Type=simple
Environment=HOME=/var/lib/asterisk
WorkingDirectory=/var/lib/asterisk
User=asterisk
Group=asterisk
ExecStart=/usr/sbin/asterisk -f -C /etc/asterisk/asterisk.conf
ExecStop=/usr/sbin/asterisk -rx 'core stop now'
ExecReload=/usr/sbin/asterisk -rx 'core reload'
LimitNOFILE=8192:8192
Restart=always

PrivateTmp=true

[Install]
WantedBy=multi-user.target

Don’t forget to re-read settings by systemd:

systemctl daemon-reload

… and now it’s time to restart Asterisk:

systemctl restart asterisk.service

Well, at this point we’ve changed the max open files limit for the Asterisk process manually and changed the Systemd Asterisk config, hoping that it will start now with the 8192 max open files limit.

If you’re interested in how many files are currently opened by your Asterisk, you may try:

voip-b2b01 ~ # lsof -p `pidof asterisk` | wc -l
956

Bash: Target Directory

October 17th, 2016

A nice article about target directory, moving and copying files/directories.
http://www.gnu.org/software/coreutils/manual/html_node/Target-directory.html#Target-directory

Asterisk 13: chan_sip sip_write Can’t send 10 type frames with SIP write

October 11th, 2016

Centos 7
Asterisk 13.1

Noticed a WARNING which filled Asterisk’s messages logfile:
bridge_simple_vs_bridge_softmix

To get rid of it, use bridge_softmix.so module instead of bride_simple.so.
Thanks to Lenar Daminov from our team for research.

UPD: https://community.asterisk.org/t/bridge-simple-and-bridge-softmix-difference/69193

jcolp, Asterisk developer:

The bridge_softmix module is more heavy weight as it gets data from each channel, transcodes it into signed linear, mixes it at an interval, and then sends it out. Even for 2 channel bridges it will do this. The bridge_simple module simply exchanges data back and forth without mixing. Frame type 10 is comfort noise which is not currently supported by Asterisk.

Asterisk 11: some required modules

July 18th, 2016

Asterisk 11, modules, which have to be loaded for other basic modules to work properly.

preload => res_http_websocket.so	; for chan_sip.so load
preload => res_speech.so		; for res_agi.so load
preload => res_agi.so			; for app_stack.so load
preload => res_monitor.so		; for app_queue.so correct load
preload => res_ael_share.so		; for pbx_ael.so correct load // also works without this,
                                        ; but WARNING while loading

Linux: limit CPU usage by running process

July 18th, 2016

Use cpulimit for this.

We have a 8-core CPU (it means the total CPU usage can be 800%, and 100% for 1 core respectively).
Our process is ‘raid-check’.

/usr/bin/cpulimit --exe /usr/sbin/raid-check --limit 600

I added this line to crontab config. The reason is that my Asterisk stops accepting calls when the CPU load is too high. Of course you can also use asterisk.conf settings to achieve the desired results.


[root@voip-ge ~]# cat /etc/cron.d/raid-check
# Run system wide raid-check once a week on Sunday at 1am by default
0 1 * * Mon root /usr/sbin/raid-check

# limit the CPU usage by executable name
1 1 * * Mon root /usr/bin/cpulimit --exe /usr/sbin/raid-check --limit 600

SSH port forwarding

July 16th, 2016

Assuming, MySQL is running on ‘remoteserver’ on 127.0.0.1:3306, and you have only SSH access to the remote server.

lexus@lexus:~$ ssh user@remote-server -L 3306:127.0.0.1:3306 -N
support@remote-server’s password: (press Ctrl-Z)
^Z
[1]+ Stopped ssh user@remote-server -L 3306:127.0.0.1:3306 -N
lexus@lexus:~$ bg
[1]+ ssh user@remote-server -L 3306:127.0.0.1:3306 -N &
lexus@lexus:~$ mysql -h 127.0.0.1 -p -u sqluser
Enter password:

Ubuntu: old releases repositories

July 8th, 2016

If your Ubuntu is too old and aptitude stopped working, showing 404 not found.

Add this to /etc/apt/sources.list, changing the CODENAME to your Ubuntu version.


## EOL upgrade sources.list
# Required
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-security main restricted universe multiverse

# Optional
#deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-backports main restricted universe multiverse

That’s enough.
Read more if you’re interested in.
https://help.ubuntu.com/community/EOLUpgrades
http://old-releases.ubuntu.com/releases/

Asterisk: ODBC configuration files relations

June 6th, 2016

https://wiki.asterisk.org/wiki/display/AST/Getting+Asterisk+Connected+to+MySQL+via+ODBC
https://wiki.asterisk.org/wiki/display/AST/Configuring+res_odbc

odbcinst.ini
===================

[MySQL]
Description = ODBC for MySQL
Driver = /usr/lib/odbc/libmyodbc.so
Setup = /usr/lib/odbc/libodbcmyS.so
FileUsage = 1

odbc.ini
===============

[asterisk-connector]
Description	= MySQL connection to 'asterisk' database
Driver = MySQL
Database = asterisk
Server = localhost
Port = 3306
Socket = /var/lib/mysqld/mysqld.sock

res_odbc.conf
====================

[asterisk]
enabled => yes
dsn => asterisk-connector	// points to DB connection in odbc.ini
username => asterisk
password => welcome
pooling => no
limit => 1
pre-connect => yes

cdr_adaptive_odbc.conf
==============================

connection = asterisk // The database connection to be used. This is a reference 
                      // to the configured connection in res_odbc.conf. This field is required.
table = // The table name. This field is required.

AcmePacket: go on rejecting!

May 19th, 2016

One more post about INVITE rejecting.

The task: reject malicious SIP traffic coming from some country to our number. All INVITEs contain a From: header with 12-digits number starting with 666. The may also contain a plus sign at the beginning or 810, or +810.

It is also important to set a ‘new-value’ parameter, containing a status code and SIP description (in form of “Code:Description”), as some (or maybe most) PBXses/softswitches/proxies go on sending INVITEs if we just do ‘action reject’. After answering from AcmePacket with something like “403 Forbidden” the remote side stops sending endless INVITEs to AcmePacket.

acme_reject_with_prefix

Part of sip-manipulation:

        header-rule
                name                                    dropHACKERS
                header-name                             From
                action                                  manipulate
                comparison-type                         pattern-rule
                msg-type                                any
                methods                                 INVITE
                match-value                             
                new-value                               
                element-rule
                        name                                    dropHACKERS1
                        parameter-name                          From
                        type                                    uri-phone-number-only
                        action                                  reject
                        match-val-type                          any
                        comparison-type                         pattern-rule
                        match-value                             666[0-9]{9}$
                        new-value                               403:Forbidden

This is how it looks like after rejecting malicious INVITE with “403 Forbidden”:
acme_reject_with_prefix2

SQL: get the database size

March 25th, 2016

I’m going on writing primitive posts about the fascinating world of relational database management systems. :)

MySQL, size of all databases:

mysql> SELECT table_schema \
       "Database name", \
       sum( data_length + index_length ) / 1024 / 1024 \
       "Data Base Size in MB" FROM \
       information_schema.TABLES GROUP BY table_schema;
+--------------------+----------------------+
| Database name      | Data Base Size in MB |
+--------------------+----------------------+
| fluxbb             |           0.04585648 |
| information_schema |           0.00781250 |
| mysql              |           0.60614872 |
| wordpress          |           2.22493362 |
+--------------------+----------------------+
4 rows in set (0.13 sec)

PostgreSQL, size of all databases:

cdr=> SELECT pg_database.datname,pg_size_pretty(pg_database_size(pg_database.datname)) \
      AS size FROM pg_database;
  datname  |  size   
-----------+---------
 template1 | 6705 kB
 template0 | 6697 kB
 postgres  | 6820 kB
 cdr       | 28 GB

Size of ‘acme_cdr’ table:

cdr=> SELECT pg_size_pretty(pg_total_relation_size('acme_cdr'));
 pg_size_pretty 
----------------
 28 GB