Table of Contents

Introduction and requirements

Configuring traffic shaping in Linux could be quite complex task. The main goals of NiceShaper are to be intuitive and flexible. This approach is achieved by simplifying things which are not easy in HTB, it means giving the ability to use the selected subset of QOS features which are the most important to have available. However, also adding additional features.

For sure, it's possible to create an entry configuration without reading this documentation document using example configuration files included in the program package. At least solid knowledge about IP networks is rather must to have. So, let's adjust internet access parameters, in /etc/niceshaper/config.conf file, and collect the hosts or classes list in /etc/niceshaper/class.conf file, then execute the niceshaper start command. It's done! Just so simple to start using NiceShaper. Use the niceshaper status command to observe how it works. But, keep in mind, after you get the first look onto the examples, then it's worth to look on documentation unless you are not interesting in another features. Still, don't be fear when you do not understand everything here as it's far beyond of your needs for the beginning.

In minimal configuration you need the router with your favourite Linux distribution. Iptables tool installed. HTB, SFQ packet scheduling algorithms, U32 and FW kernel filters compiled into the kernel. Hereinafter, HTB class means HTB packet scheduling class while NiceShaper class works on top of HTB class. Queuing algorithms means SFQ and the other classless queueing disciplines. Kernel filters means U32 and FW packets classifiers while NiceShaper filter works, again, on top of such kernel filters and/or iptables rules. In most popular Linux distributions HTB, SFQ, U32, and FW are by default compiled into the kernel or available as modules. Unfortunately, IMQ support is unusual to be compiled into the iptables and kernel.

For purpose of this documentation we assume that we have a router equipped with two interfaces. WAN is connected to the eth0 interface and LAN to the eth1. Example public IP address of the router is 198.51.100.100 and LAN network is 192.168.0.0/24.

NiceShaper uses egress shaping approach for traffic shaping. Therefore, it's fundamental that classes which control the forwarded traffic have to be placed on the interface which is outbound for such traffic. So, in assumed environment, shaping of download traffic has to be placed on eth1, but shaping of upload traffic has to be placed on eth0. Why? Because, for above assumptions, downloaded packets incomes into the router via eth0 interface and then are sent to the local network via eth1 interface. It makes the eth1 the outbound interface for download. Explanation for opposite direction is analogical.

Proper choose of outbound interfaces is crucial for configuration of NiceShaper hosts, classes, and several directives. For example, if LAN network is hidden behind NAT then it's needed to shape upload traffic using packet marking method. In this case, it's needed to put the egress interface (eth0) into the mark-on-iface directive. Additionally, IMQ interfaces as a workaround method for NAT problem are supported as well. Both, packets marking and IMQ interfaces are documented in hereafter.

Once again, these example interfaces are valid for previous assumptions but not necessarily for your router.

Words about interfaces notation. Neither Iptables nor Iproute understand concept of the network interface aliases. Thus colon and alias number should be omitted from the configuration. It's completely safe. However, despite the fact that aliases does not matter, VLANs have to be indicated. The ethX.vid notation for tagged VLAN sub-interfaces is supported. While complementing about the VLANs in Linux, it's good to know that if you run traffic shaping on tagged sub-interface it's strongly suggested to completely abandon on the untagged main interface. Otherwise, you may find that kernel queues the tagged data frames twice. Once on tagged sub-interface and second time on physical one.

NiceShaper has good scalability on routers that forward traffic for up to more or less half thousand of hosts although there are notices about successful working for even thousand of hosts. In bigger networks hash tables should be used. Unfortunately, hash tables are unsupported so far.

Installation

Compilation dependencies are c++ compiler from gcc package, C and C++ standard libraries, and make utility. Unpacked package has to be compiled omitting configure step because NiceShaper is Linux only software, thus compilation procedure is simplified.
$ bunzip2 niceshaper-%{version}.tar.bz2
$ tar xf niceshaper-%{version}.tar
$ cd niceshaper-%{version}
$ make
$ su
# make install
Make install command creates all needed directories (/etc/niceshaper, /var/lib/niceshaper, and /usr/share/doc/niceshaper). It copies the compiled binary to the /usr/local/bin directory. However, the binary path may be changed using BINDIR environment variable. Example configuration files are copied to the /etc/niceshaper directory. If configuration files are already found, in target location, then files are copied with "-dist" postfix. The documentation and the rest of package content is copied to /usr/share/doc/niceshaper.

Configuration

Configuration files syntax

Required configuration files are config.conf and class.conf placed in /etc/niceshaper directory.
include file path - Used in these files allows including next one. This directive accepts absolute (with starting slash char) and relative paths (by default to /etc/niceshaper/).
Configuration consists the directives and the parameters which order is free and syntax is shown below:
parameter value [value] - Parameter with one or number of values separated with white spaces, for example:
directive parameter value [parameter value] - It's a directive with one or number of parameter-value pairs, for example:
section speed 20Mb/s shape 19Mb/s
Configuration parser splits values or parameter-value pairs into a separated lines, so depending on own preferences these examples can be written in another way:
Exceptions are: host, class, match, section headers, and macro headers. Their syntax is described later in documentation. In addition to differences in syntax, directives are also divided according to the scope of operation: directives of global section, directives of functional sections, and directives of classes file. When it comes to the directives and parameters of classes, all, except for the class header and filters, can be placed in the functional sections configurations, thus effectively providing default values for all classes contained in such section.

The basic units for rate and capacity is b/s (bits per second) and B/s (Bytes per second). The prefixes are k (kilo) and M (mega). The suffix '/s' is not compulsory and the distinction between rate and the amount of transferred data is based on the context of use. Default unit for bandwidth is b/s (bits per second).

NiceShaper provides some special chars: Char "#" (hash) is using for comment out the rest of line. Comment block is "<# comment #>" and is using to affect some piece of configuration line. Commented configuration is useless. Char ";" means end of line, it gives you way to minimize length of configuration files by write a number of semicolon separated directives in one line.

Each configuration modification requires NiceShaper to be restarted.

Remember, all examples are only examples and might not be optimal for you, although they are good point to start working with NiceShaper.

Main configuration file - with example

Main configuration file, config.conf, is divided into the sections. First required section is named global. Additionally needed section, in practice two, and possibly even more, are functional sections. By design, functional section reflects one direction of traffic on one certain WAN connection connected to the router. So, each of WAN connections needs 2 functional sections in order to make complete traffic control. Each functional section includes it's own configuration, list of NiceShaper classes, and defaults for these classes.

Functional section may contain classes that work on various interfaces. Classes contained in any number of functional sections may be placed on a certain interface (only if configured to shaping traffic that flows in the same direction).

Main configuration file example:

Global section directives and parameters:

Functional section directives and parameters:

i
Any of the classes parameters can be placed in functional section configuration, thus becomes to be defaults for all of classes contained in such section.

Hosts and classes in NiceShaper

File /etc/niceshaper/class.conf holds the list of hosts and/or classes definitions with their configuration. We can simply assume that NiceShaper class is an extension on top of HTB class. NiceShaper class has to be built by a header, at least one filter and possibly additional options. Corresponding HTB class is created when NiceShaper notices activity, afterwards removed after a certain period of inactivity. Effectively, only for active NiceShaper classes the HTB classes coexists at the same time what gives the optimal rate value of the HTB classes.

Each packet which leaves the network interface is classified to the first matching class using filters. It is vitally important to define classes for entire traffic. If not, there occurs the traffic that is not monitored and not properly shaped. In the other words, if one or more of local LAN hosts is not assigned to the classes, these hosts falls beyond the section shape value.

Hosts:

NiceShaper classes are advanced and flexible tool, but simple directive named 'host' is in most cases sufficient and clear way to create the fairly traffic shaping for all of common hosts in local network. It's the one of elements thanks to which NiceShaper offers such comfort. Host directive gives a method to configure traffic shaping in NiceShaper just by write down the list of all LAN hosts' IP addresses and assign names to them.

To define host there is needed the list of sections in which host must be placed paired with interfaces on which traffic shaping occurred, followed by IP address, and assigned name. Host directive may be placed into one or more of running sections. Sections and interfaces values are just copied from auto-hosts directive, thus it's not needed to repeat them within each used host.

It can be assumed that host directive is some kind of NiceShaper macro. NiceShaper automatically translates the host directive into the classes and filters.

NiceShaper host definition is placed in one line, using the syntax as given below:

host ip name
For example:
These host directives will be translate, by NiceShaper configuration parser, to these classes:
Tests dstip or srcip are used depending on section mode.

Host is pointed out by IP address only. When another filtering tests or overwriting the default parameters are needed then classes have to be used instead of hosts.

Class structure:

NiceShaper, in the current version, provides 4 types of classes. These are the standard class and the classes for special purposes: virtual, wrapper, and do-not-shape. Virtual class is described in the "Virtual class type" and the last two types in the "Router's self generated traffic shaping".

Class definition starts with the class header and ends with the next class header.

Where:

Each class must contains one or more filters:
match test <test> [test <test>]
More than one test can be joined in one filter to be more specific on assign traffic to classes. Classes can contain number of filters to allow aggregating into one class, more than one host's traffic or whatever pointed out by test.

Filters applies to packets outgoing from the interface of class (egress shaping). Kernel filters are created when NiceShaper starts, but Iptables rules if needed as well.

An example of the simplest class is as follows:

Semi-colon can be used instead of new line:

Class directives and parameters:

Each parameter of classes can be placed into the functional section configuration, becomes to be defaults for each of contained classes. Next, if needed, these defaults can be override in certain classes with the individual values.

Basics filtering tests:

Srcip and dstip can match single IP address or network subnet with the mask. In the second case the bits number notation (for example "/24") or dot-decimal notation (for example "255.255.255.0") is allowed. If FW kernel filter is used on the interface, via mark-on-ifaces parameter, mask can be non-continuous (for example 255.255.128.255).

Example filters:

Tests requiring packets marking on a class interface:

Iptables is equipped with a huge number of filters which are unfortunately not feasible using U32 kernel filter. Therefore, these filters require packets marking enabled using mark-on-ifaces directive. Then each packet captured and marked by iptables can be easily enqueued to the appropriate HTB class thanks to assigned mark value.

CAUTION! Some of these filters may need additional kernel and iptables features compiled in.

Macros of the class.conf file

Macros in the classes file are introduced to improve creating of a lot of classes (or any of configuration directives) if they are similar. Macro iterates in a loop and duplicates specified area. Macro is built with a header, content and closing tag. Header is built by curly brackets that contains macro name and parameters. Content is an ordinary directives and parameters. In content you put some special chars: $ (dollar) and % (percent). These chars are replaced by expected number or string. Macro closing tag is the curly brackets which contain slash inside.

NiceShaper shares 3 macro types: sequence, foreach-elem, and foreach-pair.

Sequence macro:

{sequence from to} content {/} - Sequence macro generates the ascending numeric values in the specified range and applies these in the place of dollar special character. Values "from" and "to" have to be positive integers in the range of 0 to 65535.
Example of sequence usage:
Macro generates:

foreach-elem macro:

{foreach-elem list} content {/} - Foreach-elem macro gets each element from the given, space separated, list of values and applies these in the place of dollar special char. Values can be numeric or text, separated by white spaces.
Example of foreach-elem usage:
Macro generates:

foreach-pair macro:

{foreach-pair pair_of_values [,pair_of_values]} content {/} - Foreach-pair macro gets pairs of values from the list. Pairs are separated by commas, elements in the pair are separated by white space. First element in the pair is named a key and is applied in the place of percent char. Second element in the pair is inserted in the place of dollar. Values can be numeric or text.
Example of foreach-pair usage:
Macro generates:

Packet marking

Packets marking is a procedure of assigning the virtual tag value to packet. Thanks to that it's still possible to identify packet sender after the source address becomes changed by SNAT (Masquerading). Technically, the value is assigned by iptables, maintained by the Linux kernel and recognizable by iptables and the FW kernel filters. FW kernel filters classifies packets into the appropriate HTB classes using tests based on a virtual mark value instead of the packet header.

Marking packets mostly do not require patching iptables, iproute, or kernel. NiceShaper manages all completely automatically, so it's very easy to use this mechanism. Therefore, packet marking is a good way to avoid using the IMQ interfaces, which are comfortable but required patching the operating system important components. Packets marking easily allows traffic shaping of packets uploaded from privately addressed local network.

So, if packets marking on interface is turned on, by mark-on-ifaces directive, U32 kernel filter on that interface is replaced with FW and NiceShaper assigns a unique mark value for each class working on such interface. All is done completely automatically. However, if necessary, it's possible to make intervention in this process thanks to a set-mark parameter. The value of set-mark has to be unique for each class.

Accepted mark values are in the range of 0 to 4294967295 (32bit unsigned value) written using decimal or hex (prefixed with "0x").

Using IMQ interfaces

NiceShaper supports IMQ compiled in AB mode (after NAT in PREROUTING chain and before NAT in POSTROUTING chain).

Within the configuration of NiceShaper, IMQ interfaces are handled almost the same way as physical interfaces. You can forget about their virtuality, with the exception of NiceShaper filters require to indicate a physical interface using out-iface test (or in-iface if iptables hook is changed to PREROUTING).

Example snip from class.conf file:

NiceShaper automatically redirects traffic to IMQ interface. This behaviour is configurable by iptables imq-autoredirect directive in the global section. If, for some reason, you disable the automatic redirect to the IMQ interface, you need to make it yourself for NiceShaper using iptables ... -j IMQ --todev imqX.

Triggers

Triggers is the feature which automatically changes the values of indicated class parameters when defined condition occurs. In the current version NiceShaper two trigger types are implemented: alter and quota.

Controllable parameters of the class:

Control parameters of triggers:

Triggers are class directives, thus most convenient way is defining them within the section. For example:
The quota trigger is higher prioritized than the alter trigger if both are enabled.

Counters of the quota trigger, while stopping NiceShaper, are stored in files named "section_name.quota" placed in the /var/lib/niceshaper directory. These files are updated each time while NiceShaper is stopped and also during work every 5 minutes. If file write fails, counters will be irretrievably lost!

Triggers works only for standard-class.

Shaping of router self generated traffic

Traffic generated by the router for his own needs is mostly negligibly small and often doesn't need to be shaped. But, in real world, router with Linux often serves file shares to the local network (simpler scenario) or even serves some network services on the internet (harder scenario).

Due to the fact that the traffic shaping takes place at the outbound interface of the machine, shaping incoming traffic (both on the Internet and the LAN side) is difficult and requires using of the IMQ interfaces.

To practically deal with topic, the most convenient is splitting the description into the 4 separated scenarios.

In all scenarios, it's required to use from-local or to-local test in filters. In case of using IMQ interfaces there is the additional in-iface test required for incoming traffic, or out-iface for outgoing traffic. It's important to let the NiceShaper know the physical interface at which packet arrives or leaves the router and also to distinguish WAN side from LAN side traffic.

Keep in mind that in most cases this special purpose classes should be put on the top of the list of classes.

To remind assumptions from documentation introduction, WAN is connected to eth0 and LAN is connected to eth1. Public address of the router is 198.51.100.100, second public address for services is 198.51.100.101, private address is 192.168.0.1, and private network is 192.168.0.0/24.

Traffic between the router and the local network (uplink throughput is not involved):

Traffic between a router and a local network is not expected to be shaped or shaping should be slight. This traffic shouldn't be shaped to the fact that the local Ethernet connection is mostly the fastest. Another big mistake could be accounting local traffic to the uplink traffic and summarizing into the Internet impacting traffic generated by hosts. Classes in number 1 and number 2 scenarios have to be wrapper or do-not-shape types.
Wrapper class and do-not-shape class doesn't belong to any section so their header is not the same as the standard class header:
class-wrapper|class-do-not-shape interface name

1. Router -> Localnet - Router sends the own generated traffic to the local network:

Traffic in this scenario is, for example: local file sharing server, FTP server, local IMAP, or POP3 servers, etc.

Using do-not-shape type class - traffic is local so you do not want to shape and account:

Using wrapper type class - traffic is local but we have WiFi bridge on LAN and we are afraid to overload, so we process shaping but with statically allocated bandwidth and still do not account to any section:

2. Router <- Localnet - Router is the final recipient of the traffic from the local network:

Traffic from the LAN to the router. For example, sending e-mails via the local SMTP server, place the files on local file sharing server, etc.

Using do-not-shape type class - NiceShaper use POSTROUTING chain at iptables and doesn't use ingress shaping at kernel QOS. So, there is no requirement to create any do-not-shape type classes as traffic described in this scenario is uncontrolled in default configuration.

Using wrapper type class - but, if we want to use wrapper type class (analogically to scenario #1), we need to use the IMQ device.

Traffic between the router and the Internet (used uplink throughput):

Putting Internet accessible services on the router should be avoided. If SNAT and Internet services on WAN side are used, services should use another public IP address. In this case, requirements of two public addresses is forced because there is no way to distinguish forwarded to local LAN and NATed traffic, in the PREROUTING chain of mangle table. This distinguish is possible in INPUT chain (simplifying), but it is not allowed to redirect traffic to the IMQ device from INPUT chain.

3. Router -> Internet - Router sends the own generated traffic to the Internet:

Packets generated locally while sending to the internet, for example mail service, web pages, and the other services on the router.

In this examples, two classes are created in order to separate traffic of two different services.

Using class contained within upload mode section and filter with a from-local test don't require extensive commentary here. Provide belief that traffic from the router to the internet is properly accounted.

4. Internet => Router - Router is the final recipient of the traffic from the Internet:

This scenario occurs when the router retrieves traffic from the internet, for example the web pages retrieves by a proxy server or system updates that may steal the bandwidth from clients.
Using class contained within download mode section and a filter to-local don't require extensive commentary here too.

By default, the classes that belong to download mode section put their filters in the chain targeted from POSTROUTING where our packets will never arrive, so a to-local test creates the cloned rule in the PREROUTING chain.

Advanced topics

Cooperation with iptables

In some cases NiceShaper uses iptables. In such cases, creates rules in mangle table when starting and flushes them when stopping. You should avoid this cases whenever possible, because rules for many NiceShaper classes generates additional system load. If at least one NiceShaper class in section required iptables rule, then rules are created for all NiceShaper classes in this section. What's furthermore worst, iptables rules are created for each section with the same mode parameter value.

To be precise, iptables rules are required and created in the following cases:

Reading any of the chains don't occur more often than 10 times per second. If a section is reloaded again, before expire time, the previously cached values will be used.

Cooperation with HTB

NiceShaper creates QOS framework based on HTB on each controlled interface. This framework is built like described on diagram below. It contains HTB classes, queuing algorithms and kernel filters. HTB classes build the tree hierarchy where traffic is propagated, from the top to the bottom, as you see.

NiceShaper almost doesn't use the tc program, but communicates directly with the Linux kernel using Netlink protocol (the code that provides this functionality is partially based on iproute code). It's the most efficient way to manage QOS objects.

HTB Diagram
On the bottom level of tree there are HTB classes which are the parents for NiceShaper classes which you define in configuration. These HTB classes share the full speed of the interface, but do not borrow from each other.

These special HTB classes act as:

1) Each section which works on an interface gets the own HTB class with throughput equal to the section speed value. This HTB class becomes parent for each of child HTB class contained within such section. One of child HTB classes is automatically created and named "Waiting Room". Traffic classified by filters in first step is directing into the Waiting Room. In second step, when section is reloading, filter are modified in order to directing traffic to newly created HTB class. After defined time of inactivity this newly created HTB class is removed and kernel filter directs traffic to Waiting Room again. This time of inactivity is set by hold parameter which default is 30 seconds. In this solution in one time there are only us much HTB classes as activated NiceShaper classes, thanks to that HTB class rate parameter value is optimal.

2) This HTB class as parent for do-not-shape and wrapper classes is created if on the interface works one of or both of specified class types. In case of do-not-shape class it is true only with iface do-not-shape-method safe parameter. This HTB class throughput is calculated from iface speed minus sections speed minus fallback class.

3) HTB fallback class works with whole traffic outgoing from the interface and unclassified by filters. In proper configuration this class should be idle all the time, otherwise you should looking for lacks in defined classes and filters.

Virtual class type

Virtual is the next type of classes in NiceShaper. Header of such class:
class-virtual section interface name - Class of this type has entries in iptables, but not in HTB. Virtual class is not the tool for any traffic shaping, it's useful for measuring and reporting of utilization of observed traffic. Class of this type belongs to the section, but observed traffic is ignored by the dynamic traffic shaping algorithm. Because of this feature, packets matched by virtual class need to be then matched by standard class which finally catches such traffic.
Example of such classes:
Example of the niceshaper status command results: 
dl                              ceil -      last-ceil (   last-traffic )
^g70-v-http^                                          (        424kb/s )
^g70-v-https^                                         (        608kb/s )
g70                         5000kb/s -       5000kb/s (       1032kb/s )