OpenSIPS 2.4 usrloc full-sharing cluster

October 25th, 2019

UPD: 2019-november-5. An official tutorial is available now! And some discussion on the mail list.

Draft/some notes…

2 OpenSIPS nodes without MongoDB

  • clusterer.so module – one node is configured as seed (in case of using MongoDB for usrloc, both nodes were seed)
    • if a non-seed node restarts, it gets usrloc from seed one via bin interface
    • if a seed node is restarted, we have to get all usrloc records from non-seed one using ‘opensipsctl fifo ul_cluster_sync’ command

Key settings:

loadmodule “usrloc.so”
modparam(“usrloc”, “db_url”, “mysql://opensips:pass@10.145.213.63/opensips”)
modparam(“usrloc”, “nat_bflag”, “NAT”)
modparam(“usrloc”, “working_mode_preset”, “full-sharing-cluster”)
modparam(“usrloc”, “location_cluster”, 1)
modparam(“usrloc”, “use_domain”, 1)

loadmodule “clusterer.so” # requires proto_bin.so
modparam(“clusterer”, “current_id”, 1)
modparam(“clusterer”, “db_mode”, 1)
modparam(“clusterer”, “db_url”, “mysql://opensips:pass@10.145.213.63/opensips”)


    # initial INVITE
    if(is_method("INVITE")) {
	t_on_failure("1");
	$var(lookup_flags) = "m";
	if (cluster_check_addr("1", "$si")) {
		xlog("si: $si . $rm from cluster, doing local lookup only\n");
	} else {
		xlog("si: $si . $rm from outside, doing global lookup\n");
		$var(lookup_flags) = $var(lookup_flags) + "g";
	}
	if (!lookup("location", "$var(lookup_flags)")) {
		t_reply("404", "Not Found");
		exit;
	}
	if (has_body("application/sdp")) {
	    rtpengine_offer("RTP/AVP replace-origin replace-session-connection ICE=remove");
	}

    }	# initial invite END

UPD: some useful notes from Liviu Chircu: http://lists.opensips.org/pipermail/users/2019-October/041819.html

Pay attention: according to his advice, we don’t need any data in ‘sip_addr’ columns when creating a full-sharing usrloc cluster. In case of federated cluster we have to fill them with IP addresses on which OpenSIPS nodes are listening.

Tags:
Posted in Uncategorized | Comments Closed

OpenSIPS: filter REGISTER requests based on username

July 11th, 2019

Formerly we’ve learned how to restrict access with permission.so module based on source IP addresses. Today I’ll show how to restrict access to your OpenSIPS based on usernames, being registering.

loadmodule "permissions.so" # no deps

... 

if (is_method("REGISTER")) {
  if(!allow_register("register")) {
    sl_send_reply(403, "Forbidden by permissions");
    exit;
  }

“Deny all, but …” policy – we will allow registrations of explicitly defined usernames and drop anybody else.

register.deny file:

ALL : ALL

 
register.allow file:

# this allows lexus, lexus2, lexus3 to register 
"^sip:lexus[23]?@alexeyka.zantsev.com" : ALL 

# regexp seems to be CORRECT, but for some reason lexus2 and lexus3 can not register 
# "^sip:lexus[\d]?@alexeyka.zantsev.com" : ALL

Have a look how it’s working! A good guy is being registered successfully:

good guy

While a bad guy had been kicked:

bad guy

Another solution using regex.so module from Pavel Eremin. The pros of this method is that it allows editing a txt file with usernames defined and reload regex.so module via MI interface (no restart needed).

And even one more from Răzvan Crainea.

Tags: ,
Posted in Uncategorized | Comments Closed

OpenSIPS: ratelimit with dynamically changeable value

May 13th, 2019

This note will instruct you how to protect each DID number connected to your OpenSIPS from SIP DDoS, limiting not the total amount of INVITE requests going to your OpenSIPS server, but only INVITEs to some certain RURI.

Check it out, I hope you like it!

This is useful when you have a plenty of SIP numbers (DIDs) connected to your server and each one accepts inbound calls, e.g. a call centre or a taxi ordering service, etc. And you have to check each destination and drop too much requests, without degradation of any other incoming calls.

This is a nice solution to prevent the situation seen on the graph in the previous post.

PS: clustering support is not described here.

    loadmodule "ratelimit.so"		     # no deps
    modparam("ratelimit", "window_size", 2)  # ban timeout, sec

Add start limit value to the startup_route:

    startup_route {
    	cache_store("local", "inc:rl", "8");
    }

Somewhere in the initial INVITE section:

# AntiDDoS for each inbound call
if($si !~ "^10\..*") {
    cache_fetch("local", "inc:rl", $var(rl));
        # $var(rl) invites/sec going to each $rU.
        # SBT is the most precise policy.
        if (!rl_check("pipe_$rU", "$(var(rl){s.int})", "SBT")) {          
            sl_send_reply("503", "Service Unavailable. AntiDDoS");
            xlog("L_INFO", "call $ci from $fU@$si:$sp to $oU@$Ri drp by rl");
            exit;
        };
};

Live statistics:

voip-sipgw01 opensips # opensipsctl fifo rl_list
PIPE::  id=pipe_9618688830 algorithm=SBT limit=8 counter=0
PIPE::  id=pipe_9020578345 algorithm=SBT limit=8 counter=0
PIPE::  id=pipe_9611157347 algorithm=SBT limit=8 counter=0
PIPE::  id=pipe_79190224444 algorithm=SBT limit=8 counter=0
...
PIPE::  drop_rate=581

Change the limit on the fly up to 10 INVITES to each $rU:

opensipsctl fifo cache_store local inc:rl 10

Tags: , , ,
Posted in Uncategorized | Comments Closed

OpenSIPS: batch dialogs killer one-liner

May 1st, 2019

Situation: VoIP ISP server suddenly became unavailable and you have some thousands of dialogs like this:

dialog::  ID=17580696317398
	state:: 3
	user_flags:: 0
	timestart:: 1556697836
	datestart:: 2019-05-01 13:03:56
	timeout:: 1556701436
	dateout:: 2019-05-01 14:03:56
	callid:: 7EB7F3F9-6B1E11E9-90F3C2ED-81742C80@XX.YY.168.28
	from_uri:: sip:ZZZXXX7146@XX.YY.168.28
	to_uri:: sip:NNNN902290@XX.YY.169.130
	caller_tag:: 8999291C-450
	caller_contact:: sip:ZZZXXX7146@XX.YY.168.28:5060
	callee_cseq:: 0
	caller_route_set:: <sip:XX.YY.169.130;lr=on;ftag=8999291C-450;did=dff3.a92b;nat=yes>
	caller_bind_addr:: udp:EE.FF.116.74:5060
	caller_sdp:: 
	CALLEES:: 
		callee:: 
			callee_tag:: as3d7dee61
			callee_contact:: sip:abc@EE.FF.116.62:5060;transport=udp
			caller_cseq:: 101
			callee_route_set:: 
			callee_bind_addr:: udp:EE.FF.116.74:5060
			callee_sdp::

To kill them in some seconds (not to wait when OpenSIPS will terminate them according to SIP timers), do:

for i in `opensipsctl fifo dlg_list | grep callid | grep \@XX.YY.168.28 | awk '{print $2}'` ; do opensipsctl fifo dlg_end_dlg $i ; done

In fact, the situation was as follows: VoIP software on the ISP side (we have a DID from it connected to our OpenSIPS cluster) had some problems and they suddenly started sending us a bunch of unique INVITES (with different Call-IDs) for any certain inbound call.

Very soon the total number of dialogs dramatically increased up to enormous values. And this is a theme of my next article!

Tags:
Posted in Uncategorized | Comments Closed

Bash: convert word list to one line

April 24th, 2019 
bastion@host:~$ cat pids.txt 
 1111
 2222
 3333
 4444
bastion@host:~$ echo $(cat pids.txt)
 1111 2222 3333 4444

for i in `echo $(cat killpids.txt)` ; do kill -9 $i ; done

UPD: a note with alternative commands from Liviu Chircu, OpenSIPS core developer:

1.  instead of `echo $(cat pids.txt)`, a better alternative would be:
`cat pids.txt | xargs`

2.  instead of "for i in `echo $(cat pids.txt)`;  do kill -9 $i; done",
a better option would be: `cat pids.txt | xargs kill -9`

Tags:
Posted in Uncategorized | Comments Closed

CentOS: disable /tmp/opensips_fifo cleanup

August 13th, 2018 
dpz2-pbx ~ # cat /usr/lib/tmpfiles.d/tmp.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

# Clear tmp directories separately, to make them easier to override
v /tmp 1777 root root 10d
v /var/tmp 1777 root root 30d

# Exclude namespace mountpoints created with PrivateTmp=yes
x /tmp/systemd-private-%b-*
X /tmp/systemd-private-%b-*/tmp
x /var/tmp/systemd-private-%b-*
X /var/tmp/systemd-private-%b-*/tmp

# Disable auto-remove of /tmp/opensips_fifo
x /tmp/opensips_fifo

Tags: ,
Posted in Uncategorized | Comments Closed

OpenSIPS and rtpengine

August 10th, 2018

Draft for myself

https://github.com/sipwise/rtpengine

root@debian-opensips:~/rtpengine-mr6.3.1# dpkg-checkbuilddeps
dpkg-checkbuilddeps: error: Unmet build dependencies: debhelper (>= 10~) iptables-dev (>= 1.4) libavcodec-dev (>= 6:10) libavfilter-dev (>= 6:10) libavformat-dev (>= 6:10) libavutil-dev (>= 6:10) libbcg729-dev libcurl4-openssl-dev | libcurl4-gnutls-dev | libcurl3-openssl-dev | libcurl3-gnutls-dev libevent-dev (>= 2.0) libglib2.0-dev (>= 2.30) libhiredis-dev libjson-glib-dev libpcap0.8-dev | libpcap-dev libpcre3-dev libswresample-dev (>= 6:10) libxmlrpc-c3-dev (>= 1.16.07) | libxmlrpc-core-c3-dev (>= 1.16.07) markdown

debhelper iptables-dev libavcodec-dev libavfilter-dev libavformat-dev libavutil-dev libbcg729-dev libcurl4-openssl-dev | libcurl4-gnutls-dev | libcurl3-openssl-dev | libcurl3-gnutls-dev libevent-dev libglib2.0-dev libhiredis-dev libjson-glib-dev libpcap0.8-dev | libpcap-dev libpcre3-dev libswresample-dev libxmlrpc-c3-dev | libxmlrpc-core-c3-dev markdown

Desided to install all:

root@debian-opensips:~/rtpengine-mr6.3.1# apt install debhelper iptables-dev libavcodec-dev libavfilter-dev libavformat-dev libavutil-dev libbcg729-dev libcurl4-openssl-dev libcurl4-gnutls-dev libevent-dev libglib2.0-dev libhiredis-dev libjson-glib-dev libpcap0.8-dev libpcap-dev libpcre3-dev libswresample-dev libxmlrpc-c3-dev libxmlrpc-core-c3-dev markdown
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'libcurl4-openssl-dev' instead of 'libcurl3-openssl-dev'
Note, selecting 'libcurl4-gnutls-dev' instead of 'libcurl3-gnutls-dev'
E: Unable to locate package libbcg729-dev
E: Unable to locate package libxmlrpc-c3-dev

root@debian-opensips:~/rtpengine-mr6.3.1# apt install debhelper iptables-dev libavcodec-dev libavfilter-dev libavformat-dev libavutil-dev libbcg729-dev libcurl4-openssl-dev libcurl4-gnutls-dev libevent-dev libglib2.0-dev libhiredis-dev libjson-glib-dev libpcap0.8-dev libpcap-dev libpcre3-dev libswresample-dev libxmlrpc-c3-dev libxmlrpc-core-c3-dev markdown

Download from here https://deb.sipwise.com/spce/mr6.2.1/pool/main/b/bcg729/ , instruction about this is here https://github.com/sipwise/rtpengine/tree/mr6.3.1
in ‘G.729 support’ chapter.

root@debian-opensips:~/rtpengine-mr6.3.1# wget https://deb.sipwise.com/spce/mr6.2.1/pool/main/b/bcg729/libbcg729-0_1.0.4+git20180222-0.1~bpo9+1_amd64.deb
root@debian-opensips:~/rtpengine-mr6.3.1# wget https://deb.sipwise.com/spce/mr6.2.1/pool/main/b/bcg729/libbcg729-dev_1.0.4+git20180222-0.1~bpo9+1_amd64.deb

Install
root@debian-opensips:~/rtpengine-mr6.3.1# dpkg -i libbcg729-0_1.0.4+git20180222-0.1~bpo9+1_amd64.deb
root@debian-opensips:~/rtpengine-mr6.3.1# dpkg -i libbcg729-dev_1.0.4+git20180222-0.1~bpo9+1_amd64.deb

Try to install all again:
root@debian-opensips:~/rtpengine-mr6.3.1# apt install debhelper iptables-dev libavcodec-dev libavfilter-dev libavformat-dev libavutil-dev libbcg729-dev libcurl4-openssl-dev libcurl4-gnutls-dev libevent-dev libglib2.0-dev libhiredis-dev libjson-glib-dev libpcap0.8-dev libpcap-dev libpcre3-dev libswresample-dev libxmlrpc-c3-dev libxmlrpc-core-c3-dev markdown
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libxmlrpc-c3-dev

Trying to install these pkgs:
libxmlrpc-core-c3 libxmlrpc-core-c3-dev

No such a package in Debian!(( libxmlrpc-c3-dev , trying without it…

Trying without it and get an error:

The following packages have unmet dependencies:
libcurl4-gnutls-dev : Conflicts: libcurl4-openssl-dev but 7.52.1-5+deb9u6 is to be installed
libcurl4-openssl-dev : Conflicts: libcurl4-gnutls-dev but 7.52.1-5+deb9u6 is to be installed

The conflict was because of need either gnutls or openssl. I installed gnutls:

apt install debhelper iptables-dev libavcodec-dev libavfilter-dev libavformat-dev libavutil-dev libbcg729-dev libcurl4-gnutls-dev libevent-dev libglib2.0-dev libhiredis-dev libjson-glib-dev libpcap0.8-dev libpcap-dev libpcre3-dev libswresample-dev libxmlrpc-core-c3-dev markdown

Now dpkg-checkbuilddeps shows no errors! :)
Trying to create packages:

dpkg-buildpackage

Success!!!

-rw-r--r-- 1 root root  87K июн 20 10:18 ngcp-rtpengine_6.3.1.1+0~mr6.3.1.1_all.deb
-rw-r--r--  202K  10:18 ngcp-rtpengine-daemon_6.3.1.1+0~mr6.3.1.1_amd64.deb
-rw-r--r--  934K  10:18 ngcp-rtpengine-daemon-dbgsym_6.3.1.1+0~mr6.3.1.1_amd64.deb
-rw-r--r--   33K  10:18 ngcp-rtpengine-iptables_6.3.1.1+0~mr6.3.1.1_amd64.deb
-rw-r--r--  2,3K  10:18 ngcp-rtpengine-iptables-dbgsym_6.3.1.1+0~mr6.3.1.1_amd64.deb
-rw-r--r--   52K  10:18 ngcp-rtpengine-kernel-dkms_6.3.1.1+0~mr6.3.1.1_all.deb
-rw-r--r--   80K  10:18 ngcp-rtpengine-kernel-source_6.3.1.1+0~mr6.3.1.1_all.deb
-rw-r--r--   67K  10:18 ngcp-rtpengine-recording-daemon_6.3.1.1+0~mr6.3.1.1_amd64.deb
-rw-r--r--  217K  10:18 ngcp-rtpengine-recording-daemon-dbgsym_6.3.1.1+0~mr6.3.1.1_amd64.deb
-rw-r--r--   50K  10:18 ngcp-rtpengine-utils_6.3.1.1+0~mr6.3.1.1_all.deb

And install (as recommended in manual) libavcodec-extra packages from Debian repositories.

Trying to install ‘ dpkg -i ngcp-rtpengine-kernel-dkms_6.3.1.1+0~mr6.3.1.1_all.deb’

Get and error about ‘dkms’ package abscence.

Trying to install ‘dkms’ – errors about some packages abscence. Offers to do ‘apt –fix-broken install’. I agree, and it installs many additional packages.

I try to install these created packages again:

ngcp-rtpengine-kernel-dkms_6.3.1.1+0~mr6.3.1.1_all.deb
ngcp-rtpengine-kernel-source_6.3.1.1+0~mr6.3.1.1_all.deb

ngcp-rtpengine-recording-daemon-dbgsym_6.3.1.1+0~mr6.3.1.1_amd64.deb
ngcp-rtpengine-recording-daemon_6.3.1.1+0~mr6.3.1.1_amd64.deb

ngcp-rtpengine-utils_6.3.1.1+0~mr6.3.1.1_all.deb

At this moment I check installed packages with ‘dpkg -l | grep ngcp’ and see that they are alll already installed, marked with ‘ii’, but I go on:

ngcp-rtpengine-iptables-dbgsym_6.3.1.1+0~mr6.3.1.1_amd64.deb
ngcp-rtpengine-iptables_6.3.1.1+0~mr6.3.1.1_amd64.deb

ngcp-rtpengine-daemon-dbgsym_6.3.1.1+0~mr6.3.1.1_amd64.deb
ngcp-rtpengine-daemon_6.3.1.1+0~mr6.3.1.1_amd64.deb

ngcp-rtpengine_6.3.1.1+0~mr6.3.1.1_all.deb

Maybe some initialization scripts are configured during manual installation:

root@debian-opensips:~# dpkg -i ngcp-rtpengine-daemon_6.3.1.1+0~mr6.3.1.1_amd64.deb
(Reading database ... 66096 files and directories currently installed.)
Preparing to unpack ngcp-rtpengine-daemon_6.3.1.1+0~mr6.3.1.1_amd64.deb ...
Unpacking ngcp-rtpengine-daemon (6.3.1.1+0~mr6.3.1.1) over (6.3.1.1+0~mr6.3.1.1) ...
Setting up ngcp-rtpengine-daemon (6.3.1.1+0~mr6.3.1.1) ...
Processing triggers for systemd (232-25+deb9u3) ..

Seems strange that this package hasn’t been installed as a dependency:
apt install ffmpeg

At this moment, rtpengine supports these codecs:

root@debian-opensips:~# rtpengine --codecs
PCMA: fully supported
PCMU: fully supported
G723: fully supported
G722: fully supported
QCELP: supported for decoding only
G729: fully supported
speex: fully supported
GSM: fully supported
iLBC: not supported
opus: fully supported
vorbis: fully supported
ac3: fully supported
eac3: fully supported
ATRAC3: supported for decoding only
ATRAC-X: supported for decoding only
EVRC: supported for decoding only
EVRC0: supported for decoding only
EVRC1: supported for decoding only
AMR: fully supported
AMR-WB: fully supported
PCM-S16LE: fully supported
MP3: fully supported

Very good, but I also need iLBC.

Install package:

libavcodec57

Check:
ffmpeg -decoders
ffmpeg -encoders

No success((

Found Debian packages!
Repository: http://www.deb-multimedia.org/debian-m

Add repo to our system:

deb http://mirror.yandex.ru/debian-multimedia/ stable main non-free
deb-src http://mirror.yandex.ru/debian-multimedia/ stable main non-free

Add repo’s gpg:
wget http://www.deb-multimedia.org/pool/main/d/deb-multimedia-keyring/deb-multimedia-keyring_2016.8.1_all.deb
dpkg -i deb-multimedia-keyring_2016.8.1_all.deb

apt update
apt install libilbc2 libilbc-dev

Voila!

root@debian-opensips:~# !532
rtpengine --codecs
PCMA: fully supported
PCMU: fully supported
G723: fully supported
G722: fully supported
QCELP: supported for decoding only
G729: fully supported
speex: fully supported
GSM: fully supported
iLBC: fully supported
opus: fully supported
vorbis: fully supported
ac3: fully supported
eac3: fully supported
ATRAC3: supported for decoding only
ATRAC-X: supported for decoding only
EVRC: supported for decoding only
EVRC0: supported for decoding only
EVRC1: supported for decoding only
AMR: fully supported
AMR-WB: fully supported
PCM-S16LE: fully supported
MP3: fully supported

Rtpengine minimal start command:
rtpengine -i enp0s3/10.145.213.88 -l 10.145.213.88:9876

For these WARNS in syslog:
Jun 20 17:35:13 debian-opensips rtpengine[4029]: WARNING: Failed to properly parse UDP command line '4296_1 d7:command4:pinge' from 127.0.0.1:60637, using fallback RE

found this:
https://github.com/sipwise/rtpengine/issues/266

Advice – to start with ‘–listen-ng’ option:
rtpengine -i lo/127.0.0.1 --listen-udp=127.0.0.1:12221 --listen-ng=127.0.0.1:12222
or with CLI support:
rtpengine -i enp0s3/10.145.213.88 --listen-ng=10.145.213.88:12222 --listen-cli=10.145.213.88:12555

to use it:
rtpengine-ctl -ip 10.145.213.88 -port 12555 list totals

And modify OpenSIPS config (add new port with NG):

modparam("rtpengine", "rtpengine_sock", "udp:localhost:12222")

Works!!!
# rtpengine, may depend on tm // (optional) if you want to have rtpengine_manage() fully functional
# rtpengine -i enp0s3/10.145.213.88 --listen-ng=10.145.213.88:12222
loadmodule "rtpengine.so"
modparam("rtpengine", "rtpengine_sock", "udp:10.145.213.88:12222")

Great! OpenSIPS starts without any warnings.

OpenSIPS config:

What flags to use (by Kirill Galinurov):
$var(rtpengine_flags) = "RTP/AVP replace-session-connection replace-origin ICE=remove"


 rtpengine_offer("$var(rtpengine_flags)");

My rtpengine re-writes SDP with 127.0.0.1 IP address – because I started it listening on 127.0.0.1.
But we need to start it on a non-localhost IP address.

By Kirill Galinurov:

RTP_IP[0]=external/185.128.105.15 # (m) Local IPv4/6 address for RTP. The format of the value is [NAME/]IP[!IP].
RTP_IP[1]=internal/172.20.99.247

“loopback is also possible”:

lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 185.128.105.15/32 brd 185.128.105.15 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host

To start re-writing of IP addresses in SDP, don’t follow rtpengine module docs,
but just create global onreply_route, and use there rtpengine_answer() function without flags.
After that, you’ll have IP addresses in SDP rewritten.

But still no transcoding at this stage(

	if (is_method("INVITE")) {

		lookup("location");

		if (has_body("application/sdp")) {
			#if (rtpengine_offer("RTP/AVP replace-origin replace-session-connection ICE=remove codec-mask-PCMA codec-strip-PCMU transcode-PCMU"))
			if (rtpengine_offer("RTP/AVP replace-origin replace-session-connection ICE=remove always-transcode"))
			#if (rtpengine_offer("RTP/AVP replace-origin replace-session-connection ICE=remove"))
				t_on_reply("1");
		} else {
			t_on_reply("2");
		}
	}

	if (is_method("ACK") && has_body("application/sdp"))
		rtpengine_answer();

	route(relay);



onreply_route
{
	if (has_body("application/sdp"))
		rtpengine_answer();
}

Also check: http://opensips.org/pub/events/2015-05-12_OpenSIPS-Summit_Amsterdam/Razva_Crainea-OpenSIPS_Summit2015-EdgeProxy.cfg
And comment from Razvan how to use flags: https://github.com/OpenSIPS/opensips/issues/1288#issuecomment-367293070

Tags: ,
Posted in Uncategorized | Comments Closed

editcap: cut pcap file according to time range

July 31st, 2018

To pick a part of dump from 15:55 to 15:59, do:

lexus@lexus:~$ editcap -A '2018-07-26 15:55:00' -B '2018-07-26 15:59:00' 2018-07-26_full.pcap 2018-07-26_part.pcap

Tags: , ,
Posted in Uncategorized | Comments Closed

dotdeb.org

July 12th, 2018

Up-to-date packages for Debian 8.
Never heard about this repo.
https://www.dotdeb.org

Tags: , ,
Posted in Uncategorized | Comments Closed

OpenBSD -stable upgrade

June 26th, 2018

https://unix.stackexchange.com/questions/23579/how-to-apply-updates-on-openbsd-netbsd-and-freebsd/103661#103661
http://undeadly.org/cgi?action=article&sid=20130509120042
https://stable.mtier.org/

Posted in Uncategorized | Comments Closed

« Older Entries
Newer Entries »