Running multiple instances of rtpproxy? How to set up in SER?

[Frank Durda IV]

I have reached the point where a single instance of
rtpproxy is about maxed-out, showing that its single
process is comsuming 85% or so and climbing of the CPU
available. That works out to about 600 standing
calls, the bulk of which are G.711.

This is a multi-processor/core system and I have several
other CPUs not really doing anything, so I would like to
start up multiple instances of rtpproxy, which rtpproxy
says it can do, controlling them via AF_INET sockets
instead of the default AF_UNIX.

How you set up rtpproxy for multiple instances seems
reasonably straightforward, but what you do on the SER
side to let it know that these additional copies of
rtpproxy exist and how you or SER divides call load
between them is less clear.

Is there any documentation or examples showing this
done, or better still, someone with a working ser.ctl
file who would be willing to show at least that part
of their configuration?

Ideally, I'm looking for a case where calls are
divided among rtpproxies regardless of where they
came from or where they are going, as I have one
particular call source/destination pair that is
absurdly large compared to all others, and that so
it in particular has to be distributed across
across multiple rtpproxies.

Thanks in advance!

[Valentin Nechayev]

Post by Frank Durda IV
How you set up rtpproxy for multiple instances seems
reasonably straightforward, but what you do on the SER
side to let it know that these additional copies of
rtpproxy exist and how you or SER divides call load
between them is less clear.

See changelog comment of 2005-02-24 in modules/nathelper/nathelper.c.

[Frank Durda IV]

A follow-up on using multiple instances of rtpproxy. I finally
got this to work, although I had to look at a lot of source
code and contradictory (and terse) bits of documentation and
finally experiment a while. So here is what I would
consider "real" documentation on how to use these functions
and features of both SER and rtpproxy in one place.

Although not a requirement to make this work, I'll mention
my hardware platform configuration is a Dell 2950 III
48V DC-powered unit with a single four-core processor. The
second CPU socket is empty. FreeBSD reports:

CPU: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz (2665.33-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x10676 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,
MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xce3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,
PDCM,DCA,<b19>>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4


I run rtpproxy in the two-interface mode, which from previous
messages here I have found that I must be the only person
who does, but I do have several good reasons for doing so.

I also recently reported a coding bug in rtpproxy that at
least prevents the two-interface mode from working properly.
The fix I came up for rtpproxy is:
(diff -u this time, since apparently some OSes "patch"
program can't cope with diff -c)


--- main.c.STOCK 2008-02-20 18:51:44.000000000 +0000
+++ main.c 2009-03-25 20:31:34.000000000 +0000
@@ -930,7 +940,7 @@
* cannot be trusted and address is different from one
* that we recorded update it.
*/
- if (spa->untrusted_addr == 0 && !(spa->addr[pidx] != NULL &&
+ if (spa->untrusted_addr[pidx] == 0 && !(spa->addr[pidx] !=
NULL &&
SA_LEN(ia[0]) == SA_LEN(spa->addr[pidx]) &&
memcmp(ia[0], spa->addr[pidx], SA_LEN(ia[0])) == 0)) {
rtpp_log_write(RTPP_LOG_INFO, spa->log, "pre-filling
%s's address "
@@ -940,7 +950,7 @@
spa->addr[pidx] = ia[0];
ia[0] = NULL;
}
- if (spa->rtcp->untrusted_addr == 0 &&
!(spa->rtcp->addr[pidx] != NULL &&
+ if (spa->rtcp->untrusted_addr[pidx] == 0 &&
!(spa->rtcp->addr[pidx] != NULL &&
SA_LEN(ia[1]) == SA_LEN(spa->rtcp->addr[pidx]) &&
memcmp(ia[1], spa->rtcp->addr[pidx], SA_LEN(ia[1])) == 0)) {
if (spa->rtcp->addr[pidx] != NULL)


My setup is a bit more complicated than probably typical,
so if you don't use two interfaces, simply use the one-interface
invocation of rtpproxy ("-l oneipaddr") and build up your additional
IP addresses as needed. You will see more on this in a moment.


Each rtpproxy process needs its own IP address(es) to bind to
for RTP traffic, so you will need to alias multiple IP addresses on
the same ethernet interface or use multiple interfaces. In my case,
I put multiple IP address for each side of the rtpproxy instances
on the same physical interface using "ifconfig interface-name alias"
commands.

I chose to use multiple IP addresses on the same physical interface
because I find that on this system I can't move more than about
40Mbit/sec per interface on a single rtpproxy instance before the
processor core is at 85% utilization, so four should max-out
around 160Mbit/sec, still well below the capacity of a single
gigabit ethernet interface. Depending on your CPU speeds and maker
as well as the system bus and ethernet card type you are using,
your numbers will vary. That may mean buying plug-in cards rather
than using the interfaces that happen to be on the motherboard
(which frequently are slower, interrupt intensive and generally
inefficient compared to others you can install).

The commands to put multiple IP addresses on the same interface would
be like this on BSD and similar systems: (IP addresses are altered)

ifconfig em4 10.5.6.7 netmask 255.255.255.192
ifconfig em4 alias 10.5.6.8 netmask 255.255.255.255
ifconfig em4 alias 10.5.6.9 netmask 255.255.255.255
ifconfig em4 alias 10.5.6.10 netmask 255.255.255.255

and so on. Aliases in the same netblock as the non-aliased IP address
normally have a 255.255.255.255 netmask, at least in the BSD world.
FreeBSD lets you do this in /etc/rc.conf like this:

ifconfig_em4="inet 10.5.6.7 netmask 255.255.128.0"
ifconfig_em4_alias0="inet 10.5.6.8 netmask 255.255.255.255"
ifconfig_em4_alias1="inet 10.5.6.9 netmask 255.255.255.255"
ifconfig_em4_alias2="inet 10.5.6.10 netmask 255.255.255.255"

(If you are running Linux, the way you do aliased IP addresses is
probably different.) On FreeBSD, the result looks like:

(our inside network)
ifconfig -a
...
em4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
inet 10.5.6.7 netmask 0xffffffc0 broadcast 10.5.6.63
inet 10.5.6.8 netmask 0xffffffff broadcast 10.5.6.8
inet 10.5.6.9 netmask 0xffffffff broadcast 10.5.6.9
inet 10.5.6.10 netmask 0xffffffff broadcast 10.5.6.10
ether 00:15:17:5b:xx:xx
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active

(public Internet side, built with similar commands)
em5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
inet 66.1.2.3 netmask 0xffffffc0 broadcast 66.1.2.63
inet 66.1.2.4 netmask 0xffffffff broadcast 66.1.2.4
inet 66.1.2.5 netmask 0xffffffff broadcast 66.1.2.5
inet 66.1.2.6 netmask 0xffffffff broadcast 66.1.2.6
ether 00:15:17:5b:xx:xx
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
...


Now that you have some IP addresses defined, start several instances
of rtpproxy each listening for instructions on at least a
different port than the other instances, as in:

(As found in /etc/rc.local - non BSD systems may need this elsewhere,
but this needs to be ahead of where ser is started and after the
interfaces get their IP addresses.)

/usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8001 -l 66.1.2.3/10.5.6.7
/usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8002 -l 66.1.2.4/10.5.6.8
/usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8003 -l 66.1.2.5/10.5.6.9
/usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8004 -l 66.1.2.6/10.5.6.10

CAUTION: There are probable security issues with using public IP
addresses for the rtpproxy control address, so avoid doing that.
(rtpproxy will take instructions from anybody who can get to the
port and knows how to talk to it, or might just send it junk and
crash that process. If you must use a public IP, block external
access to the port range that is used to control rtpproxy.)


Once running, reference those rtpproxy processes via a line like this
in ser.cfg:

modparam("nathelper", "rtpproxy_sock", "udp:127.0.0.1:8001=1
udp:127.0.0.1:8002=1 udp:127.0.0.1:8003=1 udp:127.0.0.1:8004=1")

That's all on one line. You will need to comment-out the existing
nathelper line that uses the AF_UNIX socket to communicate with
just the one copy of rtpproxy. If you still have the old AF_UNIX
rtpproxy running, kill it.


The "=1" part of the modparam is the weighting, and can be handy if
some of the rtpproxy processes are running on a different computer
that is faster or slower than the local one or others. If all on
the same computer, they should all be equal.

The documentation on the weighting control is unusually weak (even by
SER documentation standards), so I can't tell you exactly how the
weighting control works, what ranges of values are valid (or if
they can be floating point), or the total number of weights should
add up to a total value, or if it is simply a round-robin distribution,
where a "1" says the next call goes to this rtpproxy before advancing
to the next rtpproxy instance, while a "2" says the next two calls
go here before advancing, and so on. I'm using all "1" values for
the local system and it seems to be balancing more or less evenly,
so for other choices you will need to experiment.


Once running for a while, you should see something like this in "top"

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
2207 xxxxxxxx 1 8 0 4180K 1056K nanslp 1 17.5H 27.05% rtpproxy
2208 xxxxxxxx 1 8 0 4192K 1068K nanslp 2 17.5H 26.66% rtpproxy
2206 xxxxxxxx 1 102 0 4204K 1080K RUN 3 17.4H 25.29% rtpproxy
2205 xxxxxxxx 1 8 0 4228K 1104K nanslp 0 17.3H 19.92% rtpproxy

You should now also see in tcpdump/ngrep output that the SDP payloads
have different IP addresses in them for the rtpproxy that is handling
the given call. Based on limited experience so far, starting
more rtpproxy instances on a machine than there are total processors/cores
available doesn't seem to do anything bad, but I would think it would
be less efficient than having the numbers of rtpproxy equal to or
less than available CPUs.

I did not mess with any of the mechanisms SER offers that can force
a given call to a given instance. Such a thing might be useful
if you want to guarantee a higher level of stability in packet
forwarding or if some of the instances were functionally different.
In my case, all instances of rtpproxy have the same capabilities.


Good luck!

Frank Durda IV http://nemesis.lonestar.org
Copyright 2009, ask before reprinting.

[Iñaki Baz Castillo]

Post by Frank Durda IV
Once running, reference those rtpproxy processes via a line like this
modparam("nathelper", "rtpproxy_sock", "udp:127.0.0.1:8001=1
udp:127.0.0.1:8002=1 udp:127.0.0.1:8003=1 udp:127.0.0.1:8004=1")

Very interesting. Let me a question:

You are setting "=1" for all the RtpProxies nodes. This means that when you
call "force_rtpproxy()" it will choose one of them randomly.

In case an initial INVITE-200 is handled by RtpProxy A, and later a re-INVITE
arrives, how do you get "force_rtpproxy()" contacts the same RtpProxy A during
re-INVITE transaction?

[Frank Durda IV]

Post by Iñaki Baz Castillo
You are setting "=1" for all the RtpProxies nodes. This means that
when you call "force_rtpproxy()" it will choose one of them randomly.
In case an initial INVITE-200 is handled by RtpProxy A, and later a
re-INVITE arrives, how do you get "force_rtpproxy()" contacts the
same RtpProxy A during re-INVITE transaction?

First off, the default isn't random. Having a higher weight increases
the opportunity for calls to be assigned to a given rtpproxy, but the
choice for a given call is not random. When you run multiple
instances, you will quickly notice that some rtpproxy processes
burn more CPU than others, which can be for a variety of reasons,
including getting more calls than others because the distribution
across rtpproxy instances isn't perfectly even even if you ask for
it to be. You'll see why that happens in a moment.

Now, I take it you are asking what happens in the case of the INVITE
coming in from the calling party, and later the 183 or 200 comes back
from the called party, and for each of these a force_rtp_proxy()
is performed. How do you get both directions of the call to go to the
same rtpproxy instance, or if something else changes after the 200 OK,
even though nathelper maintains no call state? You have three options.

One, run with only one rtpproxy, which seems to be the most common
choice made. With no choices you have no worries, but you are very
limited on how many calls you can handle.

Two, force_rtp_proxy() offers the primitive capability of using the "Nn"
flag to literally hard-code a specific rtpproxy instance to go to
without regard to any other factors. This is only vaguely useful
because it means you have to force all the calls for a given
call-condition (based on some criteria, calling source IP, account
credentials, destination IP, last digit of the called number, etc)
to get some hand-balanced distribution. If you have six tiny call
sources and one huge one, it doesn't help you much, unless the
last digit trick works for you.

The limitions implicit in having the script specify the proxy to
use via the Nn flag, combined with what I consider to be the most
stupid/horrible missing bit of functionality in SER (that of not
being able to pass variables to functions/modules*), and you
are greatly limited in what smart things you can do to scale
SER+rtpproxy, if you do it via the ser.cfg file.

*I note OpenSER supposedly allows you to pass variables to functions
(at least reading their lexical scanner seems to have added the
rules to allow this, but I don't know if it actually works or not.
I would certainly use it for a large number of other things if
it was available.)


So choice One limits your ability to scale, and choice Two potentially
means a horribly complex ser.cfg file. Fortunately, there is
choice Three.

Three, SER quietly (and possibly inadvertantly) takes care of this
issue by using the Call-ID as the variable value that is used to select
the rtpproxy. The Call-ID string is ground-up in a hashing algorithm
(see select_rtpp_node() in modules/nathelper/nathelper.c) and a value
between 0 and N comes out of that, and that combined with the total
number of weight possibilities selected at start time, skipping any
disabled proxies (presumably because they became unresponsive in the
past) dictates which rtpproxy that call will be sent to. So the
distribution is not at all random nor does it do a traditional
ascending/descending orderly assignment to rtpproxy instances like
one might find in circuit assignments in TDM or CAS trunk groups,
and what I thought the behavior would be when I first read what
little documentation there was on how the weighting system worked.
The even distribution of the calls across the instances of rtpproxy
that have even weighting values still depends on how good the
hashing algorithm is, and having a good mix of incoming Call-IDs.
In practice, it will usually be somewhat off-balance, favoring one
proxy over others at any given time. Just make sure you have
enough CPU capacity so that any rtpproxy instance has room to a
little hot. If you are using an OS that allows you to lock
processes to specific CPUs, don't use that feature on rtpproxy,
unless you are giving each rtpproxy 100% of its own CPU.

So, this means that each time force or unforce rtpproxy calls,
this same hash gets performed on the same Call-ID for a given call,
and except for rare cases where a proxy has failed, you will end up
sending the force/unforce for a given Call-ID to the same rtpproxy
instance every time. At least, this is how I read the source code.

I'll point out that if the initial selection of rtpproxy from
the first force_rtp_proxy() of a given call session had simply been
recorded as an integer somewhere with the other trivia that is
maintained for the duration of a given call session, nathelper
wouldn't have to burn cycles and time recomputing the hash as
many as additional three times for the typical call (two more
force_rtp_proxy() calls for 183 and 200 responses, then an
unforce_rtp_proxy() to tear things down), but that's the limited
behavior that exists in there today.

[Valentin Nechayev]

Hi,

Post by Frank Durda IV
So, this means that each time force or unforce rtpproxy calls,
this same hash gets performed on the same Call-ID for a given call,
and except for rare cases where a proxy has failed, you will end up
sending the force/unforce for a given Call-ID to the same rtpproxy
instance every time. At least, this is how I read the source code.

Yes, that's how it works and that was the task when we were writing
this code. In absense of dialog state in SER (the most primitive one
was invented a few months later) and need to keep a characteristics
which is stable during the stable work, the only solution was a kind
of hashing by dialog ID among alive proxies. If alive proxy set is
changed, this does it best - keeps the calls which doesn't need
re-INVITE and allows them to work in most cases after re-INVITE if
used in simple scenario.

Post by Frank Durda IV
I'll point out that if the initial selection of rtpproxy from
the first force_rtp_proxy() of a given call session had simply been
recorded as an integer somewhere with the other trivia that is
maintained for the duration of a given call session, nathelper
wouldn't have to burn cycles and time recomputing the hash as
many as additional three times for the typical call (two more
force_rtp_proxy() calls for 183 and 200 responses, then an
unforce_rtp_proxy() to tear things down), but that's the limited
behavior that exists in there today.

This requires to maintain dialog state. In PortaSIP, we later moved
this to B2BUA because it already maintains dialog state and we didn't
need to extend SER which programming is extremely hard. This was
approx. version 0.9.3. If current implementation allows custom
per-transaction and per-dialog variables kept through lifetime of
according object, this would be rewritten with new mechanisms.

[Iñaki Baz Castillo]

Post by Frank Durda IV
Now, I take it you are asking what happens in the case of the INVITE
coming in from the calling party, and later the 183 or 200 comes back
from the called party, and for each of these a force_rtp_proxy()
is performed.  How do you get both directions of the call to go to the
same rtpproxy instance, or if something else changes after the 200 OK,
even though nathelper maintains no call state?  You have three options.

In fact I expected that, being transaction stateful, *SER uses the
same RtpProxy when running force_rtpproxy in request and response(s)
belongs to the same transaction. I'm not sure of it anyway.

Post by Frank Durda IV
The limitions implicit in having the script specify the proxy to
use via the Nn flag, combined with what I consider to be the most
stupid/horrible missing bit of functionality in SER (that of not
being able to pass variables to functions/modules*), and you
are greatly limited in what smart things you can do to scale
SER+rtpproxy, if you do it via the ser.cfg file.
*I note OpenSER supposedly allows you to pass variables to functions
(at least reading their lexical scanner seems to have added the
rules to allow this, but I don't know if it actually works or not.
I would certainly use it for a large number of other things if
it was available.)

yes, OpenSER allows pseudo-variable as argument in lots of functions (not all).

Post by Frank Durda IV
Three, SER quietly (and possibly inadvertantly) takes care of this
issue by using the Call-ID as the variable value that is used to select
the rtpproxy.  The Call-ID string is ground-up in a hashing algorithm
(see select_rtpp_node() in modules/nathelper/nathelper.c) and a value
between 0 and N comes out of that, and that combined with the total
number of weight possibilities selected at start time, skipping any
disabled proxies (presumably because they became unresponsive in the
past) dictates which rtpproxy that call will be sent to.

Ok, but since the chosen RtpProxy depends on the Call-ID and weight
combination, how to ensure that force_rtpproxy() in 200 selects the
same instance than the one selected by force_rtpproxy() in the INVITE?
(the same for re-INVITE/BYE).

This is, a call is established under some circumstances an these cause
a selection of a specific RtpProxy. After 30 minutes the call is put
on hold by sending a re-INVITE. Now (after 30 minutes) the
circumstances have changed so the same Call-ID causes force_rtpproxy()
to select other RtpProxy.

This basically means that the RTP session would remain open in the
first RtpProxy (until it expires?). And this also means that *SER
script cannot rely on the return code of "force_rtpproxy(l)" that is
very useful to check if the current in-dialog INVITE belongs to a call
for which initially force_rtpproxy() took place.

Post by Frank Durda IV
So, this means that each time force or unforce rtpproxy calls,
this same hash gets performed on the same Call-ID for a given call,
and except for rare cases where a proxy has failed, you will end up
sending the force/unforce for a given Call-ID to the same rtpproxy
instance every time.  At least, this is how I read the source code.

I believe that if the decision jsut depends on the Call-ID, but since
it also depends on the weight... perhaps I miss something?

Post by Frank Durda IV
I'll point out that if the initial selection of rtpproxy from
the first force_rtp_proxy() of a given call session had simply been
recorded as an integer somewhere with the other trivia that is
maintained for the duration of a given call session,

It could be added as parameter to the Record-Route header (obviously
loose routing is required if we want the proxy to manage the RTP
stream via RtpProxy).

This is:


- In the initial INVITE force_rtpproxy() sets some variable
($rtpproxy_instance = 3).

- After it, the *SER script adds a parameter to Record-Route:
Record-Route: <sip:1.2.3.4:5060>;rtpproxy_instance=1

- Later when processing the 200 (or ACK if it has SDP), re-INVITE or
BYE, the *SER script reads the "rtpproxy_instance" from Route header
and set the RtpProxy according to it (in Kamailio there is a function
"set_rtpproxy").

Does it make sense?


Thanks a lot for your response.