Skip to content

Commit 4e41a3c

Browse files
author
bedogni@unimore.it
committed
Added analysis suite
1 parent 409ceb1 commit 4e41a3c

22 files changed

Lines changed: 2774 additions & 301 deletions

‎ANALYSIS_INDEX.md‎

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
# Analysis Tools Index
2+
3+
Complete analysis toolkit for SCIoT performance evaluation.
4+
5+
## Quick Navigation
6+
7+
### 🚀 Getting Started (5 minutes)
8+
- [QUICKSTART.sh](QUICKSTART.sh) - Copy-paste commands to get started
9+
- [SIMULATION_ANALYSIS_GUIDE.md](SIMULATION_ANALYSIS_GUIDE.md) - Step-by-step workflow
10+
11+
### 📊 Tools Documentation
12+
1. **Simulation Tool** → [SIMULATION_RUNNER_README.md](SIMULATION_RUNNER_README.md)
13+
- How to run simulations with 9 predefined scenarios
14+
- Results folder structure and CSV format
15+
- Configuration options
16+
17+
2. **Analysis Tool** → [ANALYSIS_README.md](ANALYSIS_README.md)
18+
- How to use analyze_simulation.py
19+
- Output file descriptions
20+
- Interpreting results and plots
21+
22+
3. **Complete Guide** → [SIMULATION_ANALYSIS_GUIDE.md](SIMULATION_ANALYSIS_GUIDE.md)
23+
- 5-step workflow from simulation to insights
24+
- How to interpret each of the 7 plots
25+
- 5 detailed analysis examples with code
26+
- Troubleshooting guide
27+
28+
4. **Reference** → [TOOLS_SUMMARY.md](TOOLS_SUMMARY.md)
29+
- Quick reference for all commands
30+
- Integration patterns for CI/CD
31+
- File locations and performance metrics
32+
- Extension points for customization
33+
34+
### 💻 Tools & Examples
35+
- [simulation_runner.py](simulation_runner.py) - Run automated simulations
36+
- [analyze_simulation.py](analyze_simulation.py) - Generate plots and statistics
37+
- [example_analysis.py](example_analysis.py) - 10 runnable examples
38+
39+
## Typical Workflow
40+
41+
```bash
42+
# 1. Run simulations (~15 minutes)
43+
python simulation_runner.py
44+
45+
# 2. Analyze results (~10 seconds)
46+
python analyze_simulation.py simulated_results/simulation_YYYYMMDD_HHMMSS/
47+
48+
# 3. View plots and statistics
49+
open simulated_results/simulation_YYYYMMDD_HHMMSS/analysis/
50+
51+
# 4. (Optional) Run custom analysis
52+
python example_analysis.py
53+
```
54+
55+
## Output Structure
56+
57+
```
58+
simulated_results/
59+
└── simulation_20251231_143022/
60+
├── baseline_inference_results.csv
61+
├── baseline_scenario_config.json
62+
├── network_delay_20ms_inference_results.csv
63+
├── ... (9 scenarios total)
64+
└── analysis/
65+
├── 01_device_vs_edge_time.png
66+
├── 02_total_inference_time.png
67+
├── 03_throughput_comparison.png
68+
├── 04_timing_distributions.png
69+
├── 05_layer_statistics.png
70+
├── 06_scenario_comparison_dashboard.png ⭐
71+
├── 07_summary_statistics.png
72+
└── summary_statistics.csv
73+
```
74+
75+
## 9 Simulation Scenarios
76+
77+
| # | Scenario | Type | Duration | Clients | Parameters |
78+
|---|----------|------|----------|---------|------------|
79+
| 1 | baseline | Control | 30s | 1 | No delays |
80+
| 2 | network_delay_20ms | Network | 30s | 1 | 20±5ms |
81+
| 3 | network_delay_50ms | Network | 30s | 1 | 50±10ms |
82+
| 4 | computation_delay_2ms | Computation | 30s | 1 | 2±0.5ms |
83+
| 5 | computation_delay_5ms | Computation | 30s | 1 | 5±1ms |
84+
| 6 | mobile_realistic | Combined | 60s | 1 | Both delays |
85+
| 7 | unstable_network | Variance | 45s | 1 | High variance |
86+
| 8 | multi_client_baseline | Multi-client | 45s | 3 | No delays |
87+
| 9 | multi_client_network | Multi-client | 60s | 3 | Network delay |
88+
89+
## Plot Overview
90+
91+
### 01: Device vs Edge Time
92+
- Shows execution time split between device and edge
93+
- Line plot of first 50 inferences per scenario
94+
- **Use for:** Understanding workload partitioning
95+
96+
### 02: Total Inference Time
97+
- Average total time per scenario with error bars
98+
- Bar chart comparing all scenarios
99+
- **Use for:** Overall performance ranking
100+
101+
### 03: Throughput Comparison
102+
- Inferences per second per scenario
103+
- Color-coded by simulation duration
104+
- **Use for:** Responsiveness metrics
105+
106+
### 04: Timing Distributions
107+
- Boxplots of execution time distributions
108+
- Shows variance and outliers
109+
- **Use for:** Consistency analysis
110+
111+
### 05: Layer Statistics
112+
- Device vs edge layer count comparison
113+
- Shows model partitioning strategy
114+
- **Use for:** Understanding split decisions
115+
116+
### 06: Scenario Dashboard ⭐
117+
- 9-metric comprehensive overview
118+
- All key metrics in one view
119+
- **Use for:** Quick overview and presentation
120+
121+
### 07: Summary Statistics
122+
- Table of all metrics
123+
- Mean ± std dev for each metric
124+
- **Use for:** Detailed comparison and reporting
125+
126+
## Key Metrics
127+
128+
### Per-Inference
129+
- `avg_device_time` - Average device execution time (seconds)
130+
- `avg_edge_time` - Average edge execution time (seconds)
131+
- `num_device_layers` - Layers executed on device
132+
- `num_edge_layers` - Layers executed on edge
133+
134+
### Scenario-Level
135+
- `total_time = device_time + edge_time`
136+
- `throughput = count / duration`
137+
- `variance = std_dev of timing`
138+
139+
## Analysis Capabilities
140+
141+
✓ Multi-scenario batch analysis
142+
✓ Statistical distributions (boxplots)
143+
✓ Variance and outlier detection
144+
✓ Performance ranking
145+
✓ Delay impact quantification
146+
✓ Multi-client analysis
147+
✓ Layer distribution analysis
148+
✓ Custom metric export
149+
150+
## Integration Examples
151+
152+
### Compare Baseline vs Optimized
153+
```bash
154+
# Run baseline
155+
python simulation_runner.py # baseline_20251231_143022
156+
157+
# Make optimizations...
158+
159+
# Run optimized version
160+
python simulation_runner.py # baseline_20251231_144000
161+
162+
# Compare results
163+
python analyze_simulation.py simulated_results/simulation_20251231_143022
164+
python analyze_simulation.py simulated_results/simulation_20251231_144000
165+
```
166+
167+
### CI/CD Integration
168+
```yaml
169+
- name: Run Performance Simulations
170+
run: python simulation_runner.py
171+
172+
- name: Analyze Results
173+
run: python analyze_simulation.py simulated_results/simulation_*/
174+
175+
- name: Upload Analysis
176+
uses: actions/upload-artifact@v2
177+
with:
178+
name: performance-analysis
179+
path: simulated_results/*/analysis/
180+
```
181+
182+
### Custom Analysis Script
183+
```python
184+
from analyze_simulation import SimulationAnalyzer
185+
186+
analyzer = SimulationAnalyzer("simulated_results/simulation_20251231_143022")
187+
188+
# Access data
189+
baseline = analyzer.data["baseline"]
190+
network = analyzer.data["network_delay_20ms"]
191+
192+
# Custom analysis
193+
print(f"Network delay impact: {(network['avg_device_time'].mean() - baseline['avg_device_time'].mean()) * 1000:.2f}ms")
194+
```
195+
196+
## Documentation Map
197+
198+
```
199+
User wants to... See...
200+
─────────────────────────────────────────────────────────────────────
201+
Get started in 5 minutes QUICKSTART.sh
202+
Understand the complete workflow SIMULATION_ANALYSIS_GUIDE.md
203+
Run simulations SIMULATION_RUNNER_README.md
204+
Use the analysis tool ANALYSIS_README.md
205+
Interpret the plots SIMULATION_ANALYSIS_GUIDE.md (section: Understanding Plots)
206+
Understand all metrics ANALYSIS_README.md (section: CSV Format)
207+
Write custom analysis code example_analysis.py
208+
Extend the tool with new plots ANALYSIS_README.md (section: Extending)
209+
Integrate into CI/CD TOOLS_SUMMARY.md (section: Integration)
210+
Quick reference TOOLS_SUMMARY.md
211+
See code examples example_analysis.py (10 examples)
212+
Troubleshoot issues SIMULATION_ANALYSIS_GUIDE.md (section: Troubleshooting)
213+
```
214+
215+
## File Locations
216+
217+
| Component | Location | Lines |
218+
|-----------|----------|-------|
219+
| Analysis Tool | analyze_simulation.py | 370 |
220+
| Example Code | example_analysis.py | 290 |
221+
| Simulation Tool | simulation_runner.py | 567 |
222+
| Main Guide | SIMULATION_ANALYSIS_GUIDE.md | 374 |
223+
| Analysis Docs | ANALYSIS_README.md | 218 |
224+
| Simulation Docs | SIMULATION_RUNNER_README.md | 152 |
225+
| Reference | TOOLS_SUMMARY.md | 297 |
226+
| Quick Start | QUICKSTART.sh | 40 |
227+
228+
## Dependencies
229+
230+
### Required
231+
- Python 3.11+
232+
- pandas >= 1.3.0
233+
- matplotlib >= 3.4.0
234+
235+
### Optional
236+
- seaborn >= 0.11.0 (better plot styling)
237+
238+
### Install
239+
```bash
240+
pip install pandas matplotlib seaborn
241+
```
242+
243+
## Performance
244+
245+
| Operation | Duration |
246+
|-----------|----------|
247+
| Run all 9 scenarios | ~15 minutes |
248+
| Generate all plots | ~10 seconds |
249+
| Load simulation results | <100ms |
250+
| Generate single plot | <2 seconds |
251+
252+
## Support & Help
253+
254+
1. **Quick questions?** → Check [TOOLS_SUMMARY.md](TOOLS_SUMMARY.md)
255+
2. **Can't interpret plot?** → See [SIMULATION_ANALYSIS_GUIDE.md](SIMULATION_ANALYSIS_GUIDE.md)
256+
3. **Want example code?** → Run [example_analysis.py](example_analysis.py)
257+
4. **Tool not working?** → See troubleshooting in [ANALYSIS_README.md](ANALYSIS_README.md)
258+
5. **Want to extend?** → See customization in [TOOLS_SUMMARY.md](TOOLS_SUMMARY.md)
259+
260+
## Last Updated
261+
- **Date:** December 31, 2025
262+
- **Status:** ✅ Complete and tested
263+
- **Version:** 1.0
264+
- **Test Data:** Available (simulation_20251231_155838)
265+
266+
---
267+
268+
**Ready to analyze performance?** → See [QUICKSTART.sh](QUICKSTART.sh)

0 commit comments

Comments
 (0)