uCasterisk

uCasterisk (you-see-Asterisk) is a set of scripts, makefiles and patches to build Asterisk for uClinux. The target is the Blackfin BF533 STAMP board, although it should be portable to other uClinux platforms.

uCasterisk in one part of a project to build a completely open telephony platform. Both the hardware and the software are freely available under the GPL.

The uCasterisk port was derived from the OpenWRT port of Asterisk developed by Brian Capouch, and uses a similar set of scripts derived from the cross compilation tool Buildroot.

There are several ports of Asterisk to embedded hardware (for example OpenWRT and Gumstix). These ports run in a regular Linux environment on embedded hardware that supports a Memory Management Unit (MMU). The difference with uCasterisk is that it runs in a uClinux environment, i.e. on embedded processors without a MMU.

This opens up the use of Asterisk on a variety of other processors; e.g. the Blackfin which has some nice features like DSP capability, low cost, and ease of interfacing to hardware.

At the time I performed the initial port in Sep 2005, the main challenge with the port was that uClinux didn't support shared libraries (at least not in the way regular Linux does) and Asterisk uses LOTs of shared libraries. I worked around this using GNU libtool (which can emulate shared libraries) and some patches to the Asterisk .so loader. The result is that all of the shared libraries are statically linked but Asterisk is tricked into thinking they are .so's. Not quite as good as the real thing (e.g. you need to shut down uCasterisk to add/remove a module), but it seems to work fine.

Since that time uClinux has added regular shared lib support so the shared library hack is now no longer necessary, and porting new versions of Asterisk to uClinux has become a little easier.

Despite the static linking, the footprint on the Blackfin is still fairly small, e.g. my current build is about 2.4M for Asterisk which includes a bunch of shared libs (without any sound files).

I have also written about porting multithreaded apps to uClinux here.

I am a mainly a hardware/DSP guy so bending the Buildroot scripts and Asterisk Makefile to my will was a stretch for me - please feel free to suggest or help with improvements.

To test uCasterisk I have been using SIPp. The information in this section might also be useful to other Asterisk newbies who would like to use SIPp for testing, as it took me a few hours to work out how to configure Asterisk and SIPp.

I actually found this really interesting as I hit a few memory management issues related to the differences between Linux and uClinux. This meant I had to research how memory management works for multi-threaded apps on uClinux. For the interest of others I have written a short article on my blog to explain the techniques I used.

I would like to thank the Blackfin kernel developers for their efforts - I have found that the 2005R4 RC2 kernel is much more stable, and elegantly kills my apps when they misbehave rather than allowing the kernel to panic. I would also like to say that the support from the Blackfin community is just great.

Initially I found that uCasterisk would fall over after a few hundred SIP calls, often crashing uClinux. However after gaining some understanding of uClinux memory management, and upgrading the Blackfin uClinux kernel to 2005R4 RC2, it is now much more stable, for example a recent test on a BF533 STAMP executed 500,000 SIP calls over a 12 hour period.

Initially, uCasterisk was configured to use 2MByte of physical memory per call, so after about 15 simultaneous calls I tended to run out of memory. Each simultaneous call causes a new thread to be spawned, and each thread was allocated its own 2MByte stack.

I spent a few days researching stack allocation on uClinux and wrote a small library to measure the amount of stack actually used, which is described here. This allowed me to fine-tune the amount of stack required and minimise the use of physical memory.

Sometime I would like to research the memory usage further to see if this can be improved, but as a first pass it is sufficient for my needs (SOHO IP-PBX with say 4 FXO ports). I suspect with some tweaking and extra memory a T1-span could be handled by the Blackfin.

Test 1 - 0.5s calls to exercise SIP and thread creation

The test configuration is:

1. SIPp is running on a regular Linux box connected to a BF533 STAMP running uCasterisk 0.1.3 over a LAN.

2. SIPp makes a call to an internal asterisk extension (numbered 2005), that answers the call and does nothing.

3. SIPp then hangs up after 0.5 of a second (500ms).

4. SIPp repeats this at a call rate of 10 calls per second.

Here is the sip.conf file I used. Declaring the host IP allows SIPp to make calls without needing to register:

[sipp]
type=friend
context=internal
host=192.168.1.20
port=6000
user=sipp
canreinvite=no
disallow=all
allow=ulaw

This is the extension logic from extensions.conf:

[internal]
; dummy extension just for testing
exten => 2005,1,Answer
exten => 2005,2,Wait(2)
exten => 2005,3,Hangup

And this is the SIPp command line I use on the Linux box used to generate calls (bf533 is the hostname of my BF533 STAMP board).

./sipp -sn uac -d 500 -s 2005 bf533 -l 10

Results:

1. Over 500,000 calls made over a 12 hour period

Test 2 - 10s calls to exercise RTP audio

The test configuration is:

1. SIPp is running on a regular Linux box connected to a BF537 STAMP running uCasterisk 0.1.3 over a LAN.

2. SIPp makes a call to an internal asterisk extension (numbered 2000), that maps to the standard Asterisk welcome IVR menu. RTP media (audio samples) are sent to SIPp, which echoes the media back to uCasterisk. Test 1 above didn't exercise the RTP side of the connection, which loads the network subsystem more heavily than just SIP transactions and better models a real call.

3. SIPp then hangs up after 10s.

4. SIPp is configured to repeats this with a maximum of 10-20 simultaneous calls.

5. all of the demo-.gsm files were replaced with their demo-.ulaw equivalents as GSM currently runs too slowly on the Blackfin to handle simultaneous calls well. The ulaw prompts are part of the Asterisk Native Sounds package. Note that only the demo-*.ulaw prompts were replaced, the Blackfin doesn't have enough RAM to fit all of the prompts in ulaw format and run uCasterisk at the same time.

The configuration is similar to Test 1 above except the following entry has been added to extensions.conf:

[internal]
.
<snip>
.
; reach standard asterisk demo
exten => 2000,1,Goto(demo,s,1)

This is the SIPp command line I use on the Linux box used to generate calls (bf537 is the hostname of my BF537 STAMP board).

./sipp -sn uac -d 10000 -s 2000 bf537 -l 10 -mp 5606

Results:

1. Ran 10 simultaneous calls for 12 hours on both BF533 and BF537 STAMPs, for example 45,000 10-second calls, 22M RTP packets.

2. The load average for the BF533 STAMP was 0.3, and only 0.08 for the BF537.

3. I am not sure why the load average for the BF537 was low compared to the BF533, however I suspect network performance. The BF537 has a built in Ethernet MAC and different Ethernet driver.

4. During the test I would periodically dial 2000 ("Welcome to Asterisk…") using a SIP IP-phone and check that the audio was clean (i.e. not breaking up).

5. The BF537 has also been run for a few hours at 20 simultaneous calls and seemed to work OK, however my current aim to to demonstrate stable performance for a 4-FXO port SOHO IP-PBX so I haven't tested or optimised the system to handle higher loads.

The Blackfin is a powerful DSP processor and is capable of running multiple speech codecs in real time.

In May 2006 I worked with Jean-Marc Valin from the Speex project to reduce the MIPs required for Speex encoding from 40 to around 23. The techniques used to optimise Speex are discussed here. Also in May 2006 I developed an optimised port of GSM for the Blackfin, which is discussed here.

Here is the translation matrix for uCasterisk 0.1.4 running on a BF537 STAMP:

*CLI> show translation
         Translation times between formats (in milliseconds)
          Source Format (Rows) Destination Format(Columns)

         g723   gsm  ulaw  alaw  g726 adpcm  slin lpc10  g729 speex  ilbc
   g723     -     -     -     -     -     -     -     -     -     -     -
    gsm     -     -     4     4     -     7     3     -     -    38     -
   ulaw     -     9     -     1     -     5     1     -     -    36     -
   alaw     -     9     1     -     -     5     1     -     -    36     -
   g726     -     -     -     -     -     -     -     -     -     -     -
  adpcm     -    12     5     5     -     -     4     -     -    39     -
   slin     -     8     1     1     -     4     -     -     -    35     -
  lpc10     -     -     -     -     -     -     -     -     -     -     -
   g729     -     -     -     -     -     -     -     -     -     -     -
  speex     -    19    12    12     -    15    11     -     -     -     -
   ilbc     -     -     -     -     -     -     -     -     -     -     -
*CLI>

This demonstrates the capability of the Blackfin for serious DSP work. It is feasible to run several codecs in real time, for example supporting 4-10 compressed VOIP calls.

There are commercial versions of G729 are available for the Blackfin, they run in about 22 MIPs (i.e. 10-20 channels could run in real time).

Project files are available on this site here and on the Blackfin site here.

1. Read the configuration section.

2. On the host PC:

   tar xvzf uCasterisk-0.1.4.tar.gz
   cd uCasterisk-0.1.4

3. In the file .config check that:

   BR2_TOOLCHAIN_DIR="/opt/uClinux/bfin-uclinux/bin"
   BR2_KERNEL_SOURCE="/opt/uClinux-dist/linux-2.6.x"

   point to your toolchain and target kernel sources.

4. Make sure that you have compiled a kernel in your uClinux-dist directory.

5. Then:

   make

   will download, patch, and make all the different packages you need. See the top level Makefile for other useful options.

   If you have already downloaded the tar-balls, place them in the dl directory before typing make. The build scripts will automatically bypass the download stage, speeding the build process.

6. To downloading and install (host):

   ./scripts/install_all `hostname`

   where hostname above is the hostname of your Blackfin STAMP target.

7. To start Asterisk (target):

   cd /var/tmp
   ./asterisk -vvvc

   The procedure above downloads to the RAM file system. The BF533 STAMPs only have 4MByte flash so asterisk with all its prompts will not fit. The hardware designs we are working on provide extra flash storage.

8. To just compile asterisk (say after making a few mods):

   make TOPDIR=$(pwd) -C package/asterisk compile

9. Then to just download asterisk:

   ./scripts/dltarget build-bfin-uclinux/asterisk-1.0.9/asterisk `hostname`

Sample conf files are stored in packages/asterisk/files on the host, and /etc/asterisk on the target. These are standard Asterisk conf files with the following configurations added at the end of each file:

sip.conf: An IpDialog SIP Phone 2 (SIP protocol)
iax.conf: An iaxComm Windows softphone (IAX2 protocol)

extensions.conf:

2003: The iaxComm soft phone
2004: The SIP phone
2005: A dummy phone that answers, waits one second and hangs up

The easiest way to test (no hardware required apart from target system) is to download a SIP or IAX soft phone and try to dial 2000 - which will run the standard Asterisk demo. You may need to modify /etc/asterisk/sip.conf or /etc/asterisk/iax.conf for your phone.

I have generated binaries for those who would like to test uCasterisk on their STAMP board without any compiling. The binary is a complete Linux image - all you need to do is download to your target using tftp and boot. It makes trying out uCasterisk on your STAMP very easy.

1. Make sure you have a u-boot release no later than December 2005. If you don't then the following steps will not work.

2. Download the binary for your hardware (533 or 537 STAMP) from here.

3. Copy the binary into your tftp directory on your host.

4. Then in u-boot:

   bf537> tftp 0x1000000 linux.537.ast
   bf537> bootm 0x1000000

   After a few seconds uClinux will boot (use linux.533.ast if you have a BF533 STAMP).

5. Set up your network using dhcpcd& or ifconfig

6. Optionally set /etc/asterisk/sip.conf or /etc/asterisk/iax.conf for your hard of soft phones

7. start uCasterisk with asterisk -vc

The make files can automatically generate patches against the standard asterisk and zaptel tar balls. Perform your edits in the build-bfin-uclinux/asterisk-1.0.9 or build-bfin-uclinux/zaptel-1.0.9.1 directory.

Then type:

make asterisk-make-patches

to generate the patch files, which are stored in the package directory.

In channels sub-dir iax2-parser.c was renamed iax2_parser.c, and iax2-provision.c was renamed iax2_provision.c to support the static linking using libtool (it's a long story). This messes up the patching, as diff thinks these files are new. Oops.

Also, in the zaptel package I renamed zaptel.c to zaptel1.c as I couldn't work out how to build a kernel module from two source files when one of them had the same name as the output .ko file. I am sure there is a better way to do this. Can anyone please tell me how?

You need the following configuration to build uCasterisk:

Toolchain

1. You need the 2006R1 RC4 Toolchain Release from this page.

2. The main file you need to install is the toolchain rpm. Optionally you can install the gcc 4.1 rpm.

3. Note this installs toolchain with 3.4.5 & 4.1.0 versions of gcc. The 4.1.0 version of gcc improves the speed of Speex by about 10% per codec instance.

uClinux-dist

1. You need the 2006R1 RC2 uClinux Release from this page.

2. The file you need to download is the uClinux-dist tar-ball. It's a 150MB monster download.

3. You must compile uClinux-dist. uCasterisk will not build unless you have compiled uClinux-dist to produce the executable images. Instructions on compiling are in the uClinux-dist release notes.

4. To compile uClinux-dist I used the following configuration:

   • Kernel Hacking - boot param - "root=/dev/mtdblock0 rw". This makes the root file system read-write which allows us to add the many files Asterisk requires.

   • BF537/BF533 STAMP as required.

   • Blackfin options - write back cache, this improves the speed of Speex by about 10% per codec instance.

   • Customize Vendor/User Settings - Flash Tools - MTD Utils switched off as it breaks the uClinux-dist compilation on my machine

   • I modified the file: uClinux-dist/vendors/AnalogDevices/BF537-STAMP/rc to change the hostname to bf537 and enable dhcpcd

U-boot

I am currently using the 2005R4 RC2 kernel and the December 2005 release of u-boot. If you use an earlier version of u-boot your kernel will not boot correctly.

Expect

Expect is installed to automate downloading to the target. If you do not have expect installed you will not be able to use the download scripts. On my host:

[david@solomon uCasterisk-0.1.4]$ expect -v
expect version 5.42.1

Stay up to date with the Blackfin uClinux kernels/toolchain - it is definitely worth it as they are improving rapidly.

These lines in uCasterisk-x.y/.config:

BR2_ARCH="bfin-uclinux"
BR2_TOOLCHAIN_DIR="/opt/uClinux/bfin-uclinux/bin"

define the architecture. I would be interested to know if uCasterisk works on other architectures.

Here is a short description of each directory. For more information on Buildroot check out the Buildroot site.

1. build-bfin-uclinux: is where each of the packages is built using the Blackfin cross compiler. The Buildroot scripts untars each package, apply patches, then builds the package under build-bfin-uclinux. This directory is created dynamically and is not part of the uCasterisk tar ball.

2. staging-bfin-uclinux: For some packages there is an option "install" step where package components are installed in staging-bfin-uclinux for use by other packages. For example when ncurses is built its header files are installed in staging-bfin-uclinux/usr/include and libncurses.a is installed in staging-bfin-uclinux/usr/lib. This directory is also created dynamically and is not part of the uCasterisk tar ball.

3. package: Contains the top level Makefile for each package that gets built for uCasterisk. Also contains patches, and configuration files. This is where most of the cleverness of the Buildroot system lies, for example the top level Makefile sets up the appropriate tool chain for the build before calling the regular package make file.

4. dl: Contains tar-balls of packages downloaded from the web. These are automatically fetched the first time the build process is run.

5. scripts: Useful scripts supporting the Buildroot process and testing on the Blackfin target hardware.

6. asciidoc: CSS stylesheets used to control AsciiDoc formatting

The build process works like this:

1. The top level Makefile calls the package Makefiles for each package.

2. Each package Makefile downloads the tar-ball, un tars it, patches it, configures and builds it, and optionally "installs" the package in the staging directory.

3. After all the dependant packages are built, eventually Asterisk gets built.

If you are interested in working on this project, here are some interesting tasks in rough order of importance:

1. The Zaptel echo canceller needs optimisation, it is currently in vanilla C and running much slower than it needs to. Some Blackfin assembly and C code optimisation is required. A little echo is also being heard on SIP calls, this may be due to the driver tx and rx sample being out of alignment by one frame. Some further investigation is needed here.

2. Integrate the SD-card drivers available for the Blackfin, for example get the Blackfin to boot uCasterisk from a SD-card, and save/replay voice mail from SD-card. Jerry Zeng from Analog Devices is doing some work on this.

3. Music on hold doesn't work, as I haven't set up a MP3 player for the Blackfin yet.

4. There are a few fork() to vfork() conversion issues that need to be resolved, for example music on hold, running uCasterisk as a daemon, and voice mail.

5. Convert uCasterisk to use shared libraries. The latest uClinux-dist release supported .so shared libraries - this makes porting larger applications (like Asterisk) from Linux to uClinux much easier. The main difference between uCasterisk and Asterisk is a hack to use static libs but make them appear like shared libraries. This hack can now be removed, which will streamline ports of Asterisk (and other large applications) to the Blackfin. ncurses and zlib now exist in the regular Blackfin uClinux-dist so don't need to be built as part of uCasterisk. NOTE: Eko Didik Widianto is working on this, he has ported a recent version of Asterisk (1.2.9) to the Blackfin, see this thread for more information.

6. The uClinux power of two memory allocation routine is currently causing problems when the number of calls increases (i.e. the system runs out of memory due to coarse way in which memory is allocated). This needs to be looked into, especially for higher densities. One solution is to allocate a large chunk of memory when asterisk starts, and allocate our internal buffers from that. Another solution is to use different system memory allocators - different schemes were available in the 2.4 kernel.

7. Merge the version of Asterisk in uClinux-dist with the latest uCasterisk code. Make the drivers capable of handing the Analog Devices 2 FXS-2FXO hardware as well as the modular analog hardware David has developed.

8. Lots of testing.