Linux: run a command with a time limit

Need to run tcpdump at night writing to a pcap file? Not a problem with a crond.
Need to stop it after 10 minutes? One more crond task initiating something like ‘ps aux | grep [t]cpd’, parsing it PID and killing it? No!

timeout 10m tcpdump -v -pnni eth0 udp src port 5060 and dst port 5060 and host 10.11.7.1 -w pbx_`date +%F_%H-%M`.pcap

timeout – a nice GNU coreutil.

SIP debugging: catch only certain types of messages

Let’s say you need to catch INVITEs only. In this case do:
ngrep -q -W byline -d eth0 INVITE\ sip

‘-W byline’ means to print each SIP packet in readable text mode, line by line
‘-q’ means to be quiet, not to print packet reception hash marks. Without this option your screen will fill up with ###### signs between captured types of packets.
‘-d eth0’ it’s clear
‘INVITE\ sip’ means show INVITEs only. Be careful: if you type ‘INVITE’ word only, you’ll catch nearly every SIP packet, as not only INVITE requests contain the word ‘INVITE’. For example a reply for OPTION request also contains this word among allowed mwthods described in the ‘Allow:’ header field.

And each INVITE request has a request-line like
INVITE sip:123@dmn.co...
I mean starting with ‘INVITE’ word, following space and following ‘sip’ word.

Asterisk 13: list of modules to be used

This list is not as short as it could be. But it still took much time to form it.


[modules]

autoload=no

preload => res_http_websocket.so

load => app_cdr.so
load => app_dial.so
load => app_mixmonitor.so
load => app_playback.so
load => app_stack.so
load => app_verbose.so
load => bridge_simple.so
load => cdr_adaptive_odbc.so
load => cdr_odbc.so
load => chan_bridge_media.so
load => chan_rtp.so
load => chan_sip.so
load => codec_a_mu.so
load => codec_g722.so
load => codec_resample.so
load => codec_ulaw.so
load => format_sln.so
load => format_wav.so
load => func_callerid.so
load => func_cdr.so
load => func_channel.so
load => func_global.so
load => func_periodic_hook.so
load => func_strings.so
load => pbx_ael.so
load => res_ael_share.so
load => res_agi.so
load => res_clialiases.so
load => res_format_attr_celt.so
load => res_format_attr_opus.so
load => res_format_attr_silk.so
load => res_format_attr_siren14.so
load => res_format_attr_siren7.so
load => res_format_attr_vp8.so
load => res_http_post.so
load => res_limit.so
load => res_odbc_transaction.so
load => res_odbc.so
load => res_rtp_asterisk.so
load => res_rtp_multicast.so
load => res_security_log.so

CentOS 7.2.1511 and Asterisk 13.1: max open files problem

Asterisk 13.1 (running under CentOS 7.2.1511, maybe some other Asterisk versions too) has a problem: setting ‘maxfiles’ in asterisk.conf does not matter.
So, one day, your Asterisk may stop processing new calls under high load, and as the max opened files limit is reached, you will even have no opportunity to enter the Asterisk CLI.

The /var/log/asterisk/messages (or Asterisk CLI, if you’ve been already logged in before reaching the max open files limit) will be filling with messages like these:

[Feb 13 16:47:00] WARNING[9283] acl.c: Cannot create socket
[Feb 13 16:47:00] WARNING[9283] rtp.c: Unable to allocate RTCP socket: Too many open files
[Feb 13 16:47:00] WARNING[14732] acl.c: Cannot create socket
[Feb 13 16:47:00] WARNING[14732] channel.c: Channel allocation failed: Can't create alert pipe! 
Try increasing max file descriptors with ulimit -n

As a workaround, you may change the soft and hard limit for the Asterisk process on the OS level. But first of all let’s check how many files are allowed for Asterisk to be opened:

voip-b2b01 ~ # cat /proc/`pidof asterisk`/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             15693                15693                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       15693                15693                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

So, you see the line with ‘1024’ value, in spite of our setting in asterisk.conf:

Max open files            1024                 1024                 files     

Let’s fire! CentOS has a nice tool called prlimit. Use it:

prlimit --pid `pidof asterisk` --nofile=8192

Now let’s check again:

voip-b2b01 ~ # cat /proc/`pidof asterisk`/limits
Limit                     Soft Limit           Hard Limit           Units     
...
Max open files            8192                 8192                 files     
...

CentOS 7.2 uses Systemd, so it’s wise to edit Asterisk-related configuration file.
We check the /etc/systemd/system/multi-user.target.wants/ directory and see an ‘asterisk.service’ file there, which is a symlink in fact, leading to /usr/lib/systemd/system/asterisk.service:

voip-b2b01 ~ # ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx  1 root root 40 июл 14 13:09 asterisk.service -> /usr/lib/systemd/system/asterisk.service
lrwxrwxrwx. 1 root root 38 сен 15  2015 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root 41 июн 28 14:44 bacula-fd.service -> /usr/lib/systemd/system/bacula-fd.service
lrwxrwxrwx. 1 root root 39 сен 15  2015 chronyd.service -> /usr/lib/systemd/system/chronyd.service
lrwxrwxrwx. 1 root root 37 сен 15  2015 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 42 сен 15  2015 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 41 июн 28 12:33 nfs-client.target -> /usr/lib/systemd/system/nfs-client.target
lrwxrwxrwx. 1 root root 39 июн 28 12:49 oddjobd.service -> /usr/lib/systemd/system/oddjobd.service
lrwxrwxrwx. 1 root root 40 сен 15  2015 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 39 сен 15  2015 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx  1 root root 40 июл 14 13:02 sendmail.service -> /usr/lib/systemd/system/sendmail.service
lrwxrwxrwx  1 root root 41 июл 14 13:02 sm-client.service -> /usr/lib/systemd/system/sm-client.service
lrwxrwxrwx. 1 root root 36 сен 15  2015 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 36 июн 28 12:49 sssd.service -> /usr/lib/systemd/system/sssd.service
lrwxrwxrwx. 1 root root 44 июн 28 14:19 zabbix-agent.service -> /usr/lib/systemd/system/zabbix-agent.service

Just uncomment/edit the parameter LimitNOFILE. And I advise you to uncomment/edit the Restart=always line to let Systemd control the Asterik is running (now you need not use Monit). You will have such a configuration:

voip-b2b01 ~ # grep -vE '^$|^;|^#' /usr/lib/systemd/system/asterisk.service 
[Unit]
Description=Asterisk PBX and telephony daemon.
After=network.target

[Service]
Type=simple
Environment=HOME=/var/lib/asterisk
WorkingDirectory=/var/lib/asterisk
User=asterisk
Group=asterisk
ExecStart=/usr/sbin/asterisk -f -C /etc/asterisk/asterisk.conf
ExecStop=/usr/sbin/asterisk -rx 'core stop now'
ExecReload=/usr/sbin/asterisk -rx 'core reload'
LimitNOFILE=8192:8192
Restart=always

PrivateTmp=true

[Install]
WantedBy=multi-user.target

Well, at this point we changed the max open files limit for the Asterisk process manually and changed the Systemd Asterisk config, hoping that it will start now with the 8192 max open files limit.

If you’re interested in how many files are currently opened by your Asterisk, you may try:

voip-b2b01 ~ # lsof -p `pidof asterisk` | wc -l
956

Bash: Target Directory

A nice article about target directory, moving and copying files/directories.
http://www.gnu.org/software/coreutils/manual/html_node/Target-directory.html#Target-directory

Asterisk 13: chan_sip sip_write Can’t send 10 type frames with SIP write

Centos 7
Asterisk 13.1

Noticed a WARNING which filled Asterisk’s messages logfile:
bridge_simple_vs_bridge_softmix

To get rid of it, use bridge_softmix.so module instead of bride_simple.so.
Thanks to Lenar Daminov from our team for research.

UPD: https://community.asterisk.org/t/bridge-simple-and-bridge-softmix-difference/69193

jcolp, Asterisk developer:

The bridge_softmix module is more heavy weight as it gets data from each channel, transcodes it into signed linear, mixes it at an interval, and then sends it out. Even for 2 channel bridges it will do this. The bridge_simple module simply exchanges data back and forth without mixing. Frame type 10 is comfort noise which is not currently supported by Asterisk.

Asterisk 11: some required modules

Asterisk 11, modules, which have to be loaded for other basic modules to work properly.

preload => res_http_websocket.so	; for chan_sip.so load
preload => res_speech.so		; for res_agi.so load
preload => res_agi.so			; for app_stack.so load
preload => res_monitor.so		; for app_queue.so correct load
preload => res_ael_share.so		; for pbx_ael.so correct load // also works without this,
                                        ; but WARNING while loading

Linux: limit CPU usage by running process

Use cpulimit for this.

We have a 8-core CPU (it means the total CPU usage can be 800%, and 100% for 1 core respectively).
Our process is ‘raid-check’.

/usr/bin/cpulimit --exe /usr/sbin/raid-check --limit 600

I added this line to crontab config. The reason is that my Asterisk stops accepting calls when the CPU load is too high. Of course you can also use asterisk.conf settings to achieve the desired results.


[root@voip-ge ~]# cat /etc/cron.d/raid-check
# Run system wide raid-check once a week on Sunday at 1am by default
0 1 * * Mon root /usr/sbin/raid-check

# limit the CPU usage by executable name
1 1 * * Mon root /usr/bin/cpulimit --exe /usr/sbin/raid-check --limit 600

SSH port forwarding

Assuming, MySQL is running on ‘remoteserver’ on 127.0.0.1:3306, and you have only SSH access to the remote server.

lexus@lexus:~$ ssh user@remote-server -L 3306:127.0.0.1:3306 -N
support@remote-server’s password: (press Ctrl-Z)
^Z
[1]+ Stopped ssh user@remote-server -L 3306:127.0.0.1:3306 -N
lexus@lexus:~$ bg
[1]+ ssh user@remote-server -L 3306:127.0.0.1:3306 -N &
lexus@lexus:~$ mysql -h 127.0.0.1 -p -u sqluser
Enter password:

Ubuntu: old releases repositories

If your Ubuntu is too old and aptitude stopped working, showing 404 not found.

Add this to /etc/apt/sources.list, changing the CODENAME to your Ubuntu version.


## EOL upgrade sources.list
# Required
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-security main restricted universe multiverse

# Optional
#deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-backports main restricted universe multiverse

That’s enough.
Read more if you’re interested in.
https://help.ubuntu.com/community/EOLUpgrades
http://old-releases.ubuntu.com/releases/