Skip to content

Commit f25e627

Browse files
CrabeDeFrancegithub-af
authored andcommitted
tests: Add network simulator and integration tests
This commit adds comprehensive integration tests and updates to the CI pipeline to run these tests automatically. It also includes a new network simulator application that can simulate various network conditions such as: - Bandwidth limitations - Packet loss - Network outages
1 parent 27bfc4c commit f25e627

19 files changed

Lines changed: 1477 additions & 6 deletions

.github/workflows/rust.yml

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,24 @@ name: Rust
22

33
on:
44
push:
5-
branches: [ master, dev ]
5+
branches: [master, dev]
66
pull_request:
7-
branches: [ master ]
7+
branches: [master]
88

99
env:
1010
CARGO_TERM_COLOR: always
1111

1212
jobs:
1313
build:
14-
1514
runs-on: ubuntu-latest
1615

1716
steps:
18-
- uses: actions/checkout@v4
19-
- name: Build
20-
run: cargo build --verbose --workspace
17+
- uses: actions/checkout@v4
18+
- name: Build
19+
run: cargo build --verbose --workspace
20+
- name: Run tests
21+
run: cargo test
22+
- name: Install behave
23+
run: sudo apt install python3-behave python3-fusepy python3-psutil
24+
- name: Run integration tests
25+
run: behave --tags=~fail features/simple.feature

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
11
/target
22
*~
33
doc/_build
4+
features/steps/__pycache__
5+
features/__pycache__

features/bandwidth.feature

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
Feature: Send simple files with limited network bandwidth
2+
3+
Scenario: Send 100MB file with max network of 100 Mb/s
4+
Given there is a limited network bandwidth of 100 Mb/s
5+
And diode is started with max throughput of 90 Mb/s
6+
When diode-file-send file A of size 100MB
7+
Then diode-file-receive file A in 5 seconds
8+
9+
Scenario: Send multiple 100MB file with max network of 100 Mb/s, 3 files received
10+
Given there is a limited network bandwidth of 100 Mb/s
11+
And diode is started with max throughput of 90 Mb/s
12+
When diode-file-send file A of size 100MB
13+
And diode-file-send file B of size 100MB
14+
And diode-file-send file C of size 100MB
15+
Then diode-file-receive file A in 5 seconds
16+
And diode-file-receive file B in 5 seconds
17+
And diode-file-receive file C in 5 seconds
18+
19+
Scenario: Ensure bandwidth is never exceeded
20+
Given network bandwidth must not exceed 1 Mb/s
21+
And diode is started with max throughput of 1 Mb/s
22+
When diode-file-send file A of size 3MB
23+
Then diode-file-receive file A in 30 seconds

features/drop.feature

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
Feature: Send simple files with network drop
2+
3+
Scenario: Send a 100K file with drop
4+
Given there is a network drop of 5 %
5+
And diode is started
6+
When diode-file-send file A of size 100KB
7+
Then diode-file-receive file A in 5 seconds
8+
9+
Scenario: Send a 200M file with drop
10+
Given there is a network drop of 5 %
11+
And diode is started with max throughput of 100 Mb/s
12+
When diode-file-send file A of size 200MB
13+
Then diode-file-receive file A in 5 seconds
14+
15+
Scenario: Send a 1M file with high drop
16+
Given there is a network drop of 40 %
17+
And encoding block size is 20000 and repair block size is 10000
18+
And diode is started with max throughput of 100 Mb/s
19+
When diode-file-send file A of size 1MB
20+
Then diode-file-receive file A in 5 seconds

features/environment.py

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# functions to be called before or after tests must put here
2+
3+
from tempfile import TemporaryDirectory
4+
import subprocess
5+
import time
6+
import os
7+
8+
# function call before any feature or scenario
9+
def before_all(context):
10+
# build all applications before running any test
11+
proc = subprocess.Popen(['cargo', 'build', '--release', '--bin', 'lidi-receive', '--bin', 'lidi-send', '--bin', 'lidi-network-simulator', '--bin', 'lidi-flood-receive', '--bin', 'lidi-receive-file', '--bin', 'lidi-send-file'])
12+
proc.communicate()
13+
14+
15+
# function called before every test : initialize context with default values
16+
def before_scenario(context, _feature):
17+
# test temp dir
18+
context.base_dir="/dev/shm/lidi"
19+
20+
if not os.path.isdir(context.base_dir):
21+
os.mkdir(context.base_dir)
22+
23+
# delete all files in folder
24+
try:
25+
files = os.listdir(context.base_dir)
26+
for file in files:
27+
file_path = os.path.join(context.base_dir, file)
28+
if os.path.isfile(file_path):
29+
os.remove(file_path)
30+
except OSError:
31+
print("Error occurred while deleting files.")
32+
33+
context.send_dir = TemporaryDirectory(dir=context.base_dir)
34+
context.send_ratelimit_dir = None
35+
context.receive_dir = TemporaryDirectory(dir=context.base_dir)
36+
context.log_dir = TemporaryDirectory(dir=context.base_dir)
37+
38+
# files metadata
39+
context.files = {}
40+
41+
# process instances
42+
context.proc_diode_receive = None
43+
context.proc_diode_send = None
44+
context.proc_diode_send_file = None
45+
context.proc_diode_send_dir = None
46+
context.proc_network = None
47+
context.proc_diode_receive_file = None
48+
49+
# processus en arrière-plan pour capture de sortie
50+
context.background_processes = []
51+
52+
# directory containing binaries
53+
context.bin_dir = "./target/release/"
54+
55+
# some possible options
56+
context.network_down_after = None
57+
context.network_up_after = None
58+
context.network_max_bandwidth = None
59+
context.bandwidth_must_not_exceed = None
60+
context.network_drop = None
61+
context.read_rate = None
62+
63+
# port configuration
64+
context.tcp_send_port = 4000
65+
context.tcp_receive_port = 6000
66+
67+
# display
68+
context.log_config_diode_receive = None
69+
context.log_config_diode_receive_file = None
70+
context.log_config_diode_send = None
71+
context.log_config_diode_send_dir = None
72+
context.log_config_network_behavior = None
73+
74+
context.lidi_config_path = context.base_dir
75+
76+
# directory containing binaries
77+
context.bin_dir = "./target/release/"
78+
79+
# setup logging configuration
80+
setup_log_config(context, context.base_dir)
81+
82+
# function called after every test : cleanup (delete temp directories & kill processes)
83+
def after_scenario(context, _scenario):
84+
# first kill processes
85+
if context.proc_diode_receive:
86+
context.proc_diode_receive.kill()
87+
if context.proc_diode_send:
88+
context.proc_diode_send.kill()
89+
if context.proc_diode_send_file:
90+
context.proc_diode_send_file.kill()
91+
if context.proc_diode_send_dir:
92+
context.proc_diode_send_dir.kill()
93+
if context.proc_network:
94+
context.proc_network.kill()
95+
if context.proc_diode_receive_file:
96+
context.proc_diode_receive_file.kill()
97+
98+
# make sure everything is killed, even throttled_fs (fuse) which uses temp directories
99+
time.sleep(1)
100+
101+
# delete temp directories
102+
context.send_dir.cleanup()
103+
context.receive_dir.cleanup()
104+
context.log_dir.cleanup()
105+
if context.send_ratelimit_dir:
106+
context.send_ratelimit_dir.cleanup()
107+
108+
109+
def build_log_config(filename, level):
110+
return f"""
111+
appenders:
112+
file:
113+
kind: file
114+
path: {filename}
115+
116+
root:
117+
level: {level}
118+
appenders:
119+
- file
120+
"""
121+
122+
def setup_log_config(context, log_dir, level="info"):
123+
context.log_config_diode_receive = os.path.join(log_dir, "log_config_diode_receive.yml")
124+
filename = os.path.join(log_dir, "diode_receive.log")
125+
with open(context.log_config_diode_receive, "w") as f:
126+
f.write(build_log_config(filename, level))
127+
f.close()
128+
129+
context.log_config_diode_send = os.path.join(log_dir, "log_config_diode_send.yml")
130+
filename = os.path.join(log_dir, "diode_send.log")
131+
with open(context.log_config_diode_send, "w") as f:
132+
f.write(build_log_config(filename, level))
133+
f.close()
134+
135+
context.log_config_diode_send_dir = os.path.join(log_dir, "log_config_diode_send_dir.yml")
136+
filename = os.path.join(log_dir, "diode_send_dir.log")
137+
with open(context.log_config_diode_send_dir, "w") as f:
138+
f.write(build_log_config(filename, level))
139+
f.close()
140+
141+
context.log_config_diode_receive_file = os.path.join(log_dir, "log_config_diode_receive_file.yml")
142+
filename = os.path.join(log_dir, "diode_receive_file.log")
143+
with open(context.log_config_diode_receive_file, "w") as f:
144+
f.write(build_log_config(filename, level))
145+
f.close()
146+
147+
context.log_config_network_behavior = os.path.join(log_dir, "log_config_network_behavior.yml")
148+
filename = os.path.join(log_dir, "network_behavior.log")
149+
with open(context.log_config_network_behavior, "w") as f:
150+
f.write(build_log_config(filename, level))
151+
f.close()
152+

features/interrupt.feature

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
Feature: Send simple files with network interrupts
2+
3+
Scenario: Send 3x100KB file with network interrupt, 2 first files lost, last one transmitted
4+
Given there is a network interrupt of 100KB after 50KB
5+
And there is a limited network bandwidth of 100 Mb/s
6+
And diode is started with max throughput of 90 Mb/s
7+
When diode-file-send file A of size 100KB
8+
And diode-file-send file B of size 100KB
9+
And diode-file-send file C of size 100KB
10+
Then diode-file-receive file C in 5 seconds
11+
12+
Scenario: Send 3x1MB file with network interrupt, 2 first files lost, last one transmitted
13+
Given there is a network interrupt of 1MB after 500KB
14+
And there is a limited network bandwidth of 100 Mb/s
15+
And diode is started with max throughput of 90 Mb/s
16+
When diode-file-send file A of size 1MB
17+
And diode-file-send file B of size 1MB
18+
And diode-file-send file C of size 1MB
19+
Then diode-file-receive file C in 5 seconds
20+
21+
Scenario: Send 3x100MB file with network interrupt, 2 first files lost, last one transmitted
22+
Given there is a network interrupt of 100MB after 50MB
23+
And there is a limited network bandwidth of 100 Mb/s
24+
And diode is started with max throughput of 90 Mb/s
25+
When diode-file-send file A of size 100MB
26+
And diode-file-send file B of size 100MB
27+
And diode-file-send file C of size 100MB
28+
Then diode-file-receive file C in 5 seconds
29+

features/mtu.feature

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
Feature: Send simple files with limited network bandwidth
2+
3+
Scenario: Send 100MB file with MTU 9000
4+
Given diode is started with max throughput of 100 Mb/s and MTU 9000
5+
When diode-file-send file A of size 100MB
6+
Then diode-file-receive file A in 5 seconds
7+
8+
Scenario: Send multiple 100MB file with MTU 9000, 3 files received
9+
Given diode is started with max throughput of 100 Mb/s and MTU 9000
10+
When diode-file-send file A of size 100MB
11+
And diode-file-send file B of size 100MB
12+
And diode-file-send file C of size 100MB
13+
Then diode-file-receive file A in 5 seconds
14+
And diode-file-receive file B in 5 seconds
15+
And diode-file-receive file C in 5 seconds
16+

features/multiple.feature

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
Feature: Send simple small files at the same time
2+
3+
# rate limit is not necessary because when using send-file
4+
# with multiple files at once, it will send all packets
5+
# in a single connexion instead of a single file per connection
6+
Scenario: Send 10x1K file without drop
7+
Given diode is started
8+
When diode-file-send 10 files of size 1KB
9+
Then diode-file-receive all files in 5 seconds
10+
11+
Scenario: Send 10x10K file without drop
12+
Given diode is started with max throughput of 100 Mb/s
13+
When diode-file-send 10 files of size 10KB
14+
Then diode-file-receive all files in 5 seconds
15+
16+
Scenario: Send 10x100K file without drop
17+
Given diode is started with max throughput of 100 Mb/s
18+
When diode-file-send 10 files of size 100KB
19+
Then diode-file-receive all files in 5 seconds
20+

features/send_dir_basics.feature

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
Feature: Check diode-send-dir is sending one or multiple files with copy or move
2+
3+
Scenario: Copy a 1K file with diode-send-dir
4+
Given diode with send-dir is started
5+
When we copy a file A of size 1KB
6+
Then diode-file-receive file A in 5 seconds
7+
8+
Scenario: Copy multiple 1K files with diode-send-dir
9+
Given diode with send-dir is started
10+
When we copy a file A of size 1KB
11+
When we copy a file B of size 1KB
12+
When we copy a file C of size 1KB
13+
Then diode-file-receive file A in 5 seconds
14+
Then diode-file-receive file B in 5 seconds
15+
Then diode-file-receive file C in 5 seconds
16+
17+
Scenario: Move a 1K file with diode-send-dir
18+
Given diode with send-dir is started
19+
When We move a file A of size 1KB
20+
Then diode-file-receive file A in 5 seconds
21+
22+
Scenario: Move multiple 1K file with diode-send-dir
23+
Given diode with send-dir is started
24+
When we move a file A of size 1KB
25+
When we move a file B of size 1KB
26+
When we move a file C of size 1KB
27+
Then diode-file-receive file A in 5 seconds
28+
Then diode-file-receive file B in 5 seconds
29+
Then diode-file-receive file C in 5 seconds

features/send_dir_ignore.feature

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Feature: Check diode-send-dir is not sending ignored files
2+
3+
Scenario: Copy a dot file with diode-send-dir
4+
Given diode with send-dir is started
5+
When We copy a file .A of size 1KB
6+
Then diode-file-receive no file .A in 5 seconds
7+
Then file .A is in source directory
8+
9+
Scenario: Move a dot file with diode-send-dir
10+
Given diode with send-dir is started
11+
When We move a file .A of size 1KB
12+
Then diode-file-receive no file .A in 5 seconds
13+
Then file .A is in source directory
14+

0 commit comments

Comments
 (0)