Repo with Cisco T-rex traffic profiles.
-
Configure the server:
For better performance setup the following options in BIOS:- Disable Hyperthreading
- Enable 'Maximum perfomance' or 'Low latency' profile
- Enable Intel VT-d
- Enable SR-IOV (if needed)
-
Configure Linux (Debian in example):
- Increase the number of available 2MB Huge Pages:
echo "vm.nr_hugepages = 65535" > /etc/sysctl.conf && sysctl -p cd /usr/lib/x86_64-linux-gnu/ && sudo ln -s -f libc.a liblibc.a - Add the following line to the /etc/default/grub file:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" sudo update-grub reboot
- Increase the number of available 2MB Huge Pages:
-
Download and untar T-rex to /opt/trex directory:
wget --no-check-certificate --no-cache https://trex-tgn.cisco.com/trex/release/v3.02.tar.gz
sudo mkdir /opt/trex
sudo tar xvf v3.02.tar.gz -C /opt/trex --strip-components 1
- Configure Cisco T-rex interfaces:
cd /opt/trex && sudo ./dpdk_setup_ports.py -i - /etc/trex_cfg.yaml will be created.
Example:
### Config file generated by dpdk_setup_ports.py ###
- version: 2
interfaces: ['17:00.0', '17:00.1']
c: 16
port_info:
- dest_mac: b4:96:91:f6:51:95 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
src_mac: b4:96:91:f6:51:94
- dest_mac: b4:96:91:f6:51:94 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
src_mac: b4:96:91:f6:51:95
platform:
master_thread_id: 1
latency_thread_id: 24
dual_if:
- socket: 0
threads: [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]
Custom trex config (for L2 mode):
- version: 2
interfaces: ['86:00.0', '86:00.1']
prefix: trex3
zmq_pub_port: 4700
zmq_rpc_port: 4701
c: 24
limit_memory: 32768
new_memory: true
port_info:
- dest_mac: 6c:fe:54:50:8e:b9 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
src_mac: 6c:fe:54:50:8e:b8
#vlan: 1402
- dest_mac: 6c:fe:54:50:8e:b8 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
src_mac: 6c:fe:54:50:8e:b9
#vlan: 1302
platform:
master_thread_id: 26
latency_thread_id: 51
dual_if:
- socket: 1
threads: [27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50]
#threads: [41,42,43,44,45,46,47,48,49,50,51]
memory:
dp_flows: 40000000
mbuf_9k: 560000
- For enabling L3 with SR-IOV mode execute the following:
echo 4 > /sys/bus/pci/devices/0000:86:00.0/sriov_numvfs
ip link set dev ens5f0np0 vf 0 state enable
ip link set dev ens5f0np0 vf 1 state enable
ip link set dev ens5f0np0 vf 2 state enable
ip link set dev ens5f0np0 vf 3 state enable
ip link set dev ens5f0np0 vf 0 trust on
ip link set dev ens5f0np0 vf 1 trust on
ip link set dev ens5f0np0 vf 2 trust on
ip link set dev ens5f0np0 vf 3 trust on
echo 4 > /sys/bus/pci/devices/0000:86:00.1/sriov_numvfs
ip link set dev ens5f1np1 vf 0 state enable
ip link set dev ens5f1np1 vf 1 state enable
ip link set dev ens5f1np1 vf 2 state enable
ip link set dev ens5f1np1 vf 3 state enable
ip link set dev ens5f1np1 vf 0 trust on
ip link set dev ens5f1np1 vf 1 trust on
ip link set dev ens5f1np1 vf 2 trust on
ip link set dev ens5f1np1 vf 3 trust on
where 0000:86:00.0 - PCI for network card port.
Config example for L3:
- version: 2
interfaces: ['86:00.2', '86:00.3', '86:00.4', '86:00.5', '86:01.2', '86:01.3', '86:01.4', '86:01.5']
prefix: trex3
zmq_pub_port: 4700
zmq_rpc_port: 4701
c: 6
limit_memory: 32768
new_memory: true
port_info:
- default_gw : 10.255.3.254
ip : 10.255.3.100
vlan: 300
- default_gw : 10.255.4.254
ip : 10.255.4.100
vlan: 400
- default_gw : 10.255.5.254
ip : 10.255.5.100
vlan: 301
- default_gw : 10.255.6.254
ip : 10.255.6.100
vlan: 401
- default_gw : 10.255.7.254
ip : 10.255.7.100
vlan: 302
- default_gw : 10.255.8.254
ip : 10.255.8.100
vlan: 402
- default_gw : 10.255.9.254
ip : 10.255.9.100
vlan: 303
- default_gw : 10.255.10.254
ip : 10.255.10.100
vlan: 403
platform:
master_thread_id: 26
latency_thread_id: 51
dual_if:
- socket: 1
threads: [27,28,29,30,31,32]
#threads: [41,42,43,44,45,46,47,48,49,50,51]
- socket: 1
threads: [33,34,35,36,37,38]
#threads: [41,42,43,44,45,46,47,48,49,50,51]
- socket: 1
threads: [39,40,41,42,43,44]
#threads: [41,42,43,44,45,46,47,48,49,50,51]
- socket: 1
threads: [45,46,47,48,49,50]
#threads: [41,42,43,44,45,46,47,48,49,50,51]
memory:
dp_flows: 40000000
mbuf_9k: 560000
- Create systemd unit for trex:
[Unit]
Description=TREX Service Instance 1
[Service]
StandardOutput=journal
StandardError=journal
WorkingDirectory=/opt/trex
ExecStart=/opt/trex/t-rex-64 -i --astf --tso-disable --lro-disable --iom 0 --cfg /etc/trex_cfg.yaml
[Install]
WantedBy=multi-user.target
systemctl daemon-reload && systemctl start trex
Clone current repository:
git clone https://github.com/OlegKashtanov/ngfw-trex-perf.git
cd ngfw-trex-perf
pip3 install -r requirements.txt
I. Run test using Trex systemd service
sudo systemctl start trex- Start test. Example for udp_imix.py profile with 1400B frame size:
./trex_run.py -m 100 -d 60 -f trex_profiles/udp_imix.py -t ramp_up=30 frame_size_b=1400 --send_stats=false
-m: multiplier for traffic profile
-d: test duration in seconds (steady state period)
-t: profile's tunables
- Run test with attacks (will be sent in sequence):
./trex_run.py --attack-interval 1 --send_stats False -m 1 -a <pcaps_with_attacks_repo> -f trex_profiles/udp_imix.py --attacks-once
- Available options for trex_run.py:
usage: trex_run.py [-h] [-s SERVER] [--sync_port SYNC_PORT] [--async_port ASYNC_PORT] [--trex_instance TREX_INSTANCE] [-m MULT] [-f FILE] [-d DURATION] [-a ATTACKS_PATH] [--drp DRP] [-t [TUNABLES ...]] [--send_stats SEND_STATS] [--influx_addr INFLUX_ADDR]
[--influx_port INFLUX_PORT] [--influx_admin INFLUX_ADMIN] [--influx_passwd INFLUX_PASSWD] [--influx_db INFLUX_DB] [--influx_interval INFLUX_INTERVAL] [--grafana_url GRAFANA_URL]
[--latency_pps LATENCY_PPS] [--json] [--bw] [--ngfw-hostname NGFW_HOST] [--cc-dur CC_DURATION] [--attack-interval ATTACK_INTERVAL] [--capture] [--attacks-once] [--arp-retries ARP_RETRIES]
TRex ASTF mode
optional arguments:
-h, --help show this help message and exit
-s SERVER remote TRex address (default: 127.0.0.1)
--sync_port SYNC_PORT
the RPC port (default: 4501)
--async_port ASYNC_PORT
the ASYNC port (subscriber port) (default: 4500)
--trex_instance TREX_INSTANCE
Trex instance (default: None)
-m MULT multiplier of main traffic (default: 1.0)
-f FILE profile path for sending main traffic (default: None)
-d DURATION duration of traffic, sec (default: 10)
-a ATTACKS_PATH attack pcaps absolute path directory for sending attacks (default: None)
--drp DRP Allowed main traffic drops (connections), % (default: 0.1)
-t [TUNABLES ...] tunables for main profile: key1=value1 key2=value2... (default: {})
--send_stats SEND_STATS
Send stats to InfluxDB (default: False)
--influx_addr INFLUX_ADDR
InfluxDB address (default: 127.0.0.1)
--influx_port INFLUX_PORT
InfluxDB port (default: 8086)
--influx_admin INFLUX_ADMIN
InfluxDB admin user (default: admin)
--influx_passwd INFLUX_PASSWD
InfluxDB admin password (default: admin)
--influx_db INFLUX_DB
Influx DB name (default: trex)
--influx_interval INFLUX_INTERVAL
Influx send interval, sec (default: 10)
--grafana_url GRAFANA_URL
Grafana URL (default: http://127.0.0.1:3000)
--latency_pps LATENCY_PPS
ICMP packets rate (default: 0)
--json Output in json format (default: False)
--bw Get resolved ports bandwidth (default: False)
--ngfw-hostname NGFW_HOST
NGFW hostname for grafana dashboards (default: None)
--cc-dur CC_DURATION Concurrent connections steady state duration, sec (default: 0)
--attack-interval ATTACK_INTERVAL
Time interval between attacks sending, sec (default: 1.0)
--capture Capture first 1000 pckts since starting main profile (default: False)
--attacks-once Send all attacks only once (default: False)
--arp-retries ARP_RETRIES
ARP retries with 1 second interval (default: 10)