diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..d64b4ad
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,4 @@
+scripts/.ipynb_checkpoints/
+scripts/output_analysis/
+scripts/output_per_frame/
+scripts/output_scores/
diff --git a/README.md b/README.md
index cee4809..999c26f 100644
--- a/README.md
+++ b/README.md
@@ -8,26 +8,55 @@
-## 👷♀️ Requirements
+## 📊 Available tools
-Please ensure the following dependencies are installed before running scripts from this repository:
+This repository contains multiple scripts which can be run as is for reporting purposes, or serve as a baseline/guide for custom analysis scripts. Below we list all tools, with example outputs below. We also mention which command line argument should be passed to Evolve to generate the requested graphs.
+
+- **`run_on_android.py`** Launches the Evolve benchmark on a connected Android device over ADB and pulls the result files back to your machine. This script requires ADB (Android Debug Bridge) to be installed.
+- **`compare_deep_analysis.py`** CLI tool that compares two deep-analysis exports and writes the top 20 passes with the largest mean / standard-deviation differences to CSV. Requires `--export-deep-analysis`.
+- **`scores_plotter.ipynb`** Bar chart comparing the Evolve scores per metric across benchmark runs or devices. Requires `--export-scores`.
+
+ 
+
+- **`per_frame_plotter.ipynb`** Line graph of the per score bucketed execution time/metric (frame time, ray tracing, rasterization, compute, driver, …) throughout the benchmark timeline. Can be used to compare between runs. Requires `--export-per-frame`.
-- Python 3.7
-- ADB (Android Debug Bridge)
+ 
+ 
+- **`deep_analysis_plotter.ipynb`** Line graph of the execution time per render/compute/rt pass. Can be used to compare between runs. Requires `--export-deep-analysis`.
-## 📊 Comparing deep analysis output
+ 
-Using the `compare_deep_analysis.py` script located in the `scripts` directory, you can compare the results of two separate deep analysis output files in multiple ways. For the analysis methods, the scripts will first do an attempt to average over all loop iterations of the output. If you ran Evolve with `--looping 5`, each frame in the output will use the mean from each frame from each of the Evolve benchmark iterations.
+- **`frame_breakdown_stackplot.ipynb`** Stacked area chart breaking down each frame's GPU time by render/compute/rt pass (top 20 passes + other). Requires `--export-deep-analysis`.
-## Additional Requirements
-In addition to the requirements listed in [Requirements](#👷♀️-requirements), this script also requires the use of several python packages. These can be installed as follows:
+ 
+
+## 📏 Capturing data
+
+These scripts require specific custom Evolve outputs, which can be generated using the following command line arguments when launching evolve.
+
+ - `run-custom --export-scores scores.csv`
+ - `run-custom --export-per-frame per_frame.csv`
+ - `run-custom --export-deep-analysis deep_analysis.json`
+
+## 👷♀️ Requirements
+
+Please ensure the following dependencies are installed before running scripts from this repository:
+
+- Python 3.11+
+- ADB (Android Debug Bridge), only required for `run_on_android.py`
+
+The scripts and notebooks depend on several Python packages, which can be installed with:
```sh
python -m pip install -r requirements.txt
```
-## Usage
+## 📊 Using `compare_deep_analysis.py`
+
+Using the `compare_deep_analysis.py` script, you can compare the results of two separate deep analysis output files in multiple ways. For the analysis methods, the scripts will first do an attempt to average over all loop iterations of the output. If you ran Evolve with `--looping 5`, each frame in the output will use the mean from each frame from each of the Evolve benchmark iterations.
+
+### Usage
```sh
usage: Evolve Deep Analysis Comparison [-h] [--pass_mean_comparison PASS_MEAN_COMPARISON] [--pass_stdev_comparison PASS_STDEV_COMPARISON] deep_analysis_file deep_analysis_file
```
@@ -43,4 +72,4 @@ At least one of `--pass_mean_comparison` or `--pass_stdev_comparison` arguments
### Example usage
```sh
python compare_deep_analysis.py --pass_mean_comparison mean_comparison.csv --pass_stdev_comparison stdev_comparison.csv deep_analysis_gpu_1.json deep_analysis_gpu_2.json
-```
\ No newline at end of file
+```
diff --git a/docs/images/tools/deep_analysis_plotter.png b/docs/images/tools/deep_analysis_plotter.png
new file mode 100644
index 0000000..8840059
Binary files /dev/null and b/docs/images/tools/deep_analysis_plotter.png differ
diff --git a/docs/images/tools/frame_breakdown_stackplot.png b/docs/images/tools/frame_breakdown_stackplot.png
new file mode 100644
index 0000000..f765590
Binary files /dev/null and b/docs/images/tools/frame_breakdown_stackplot.png differ
diff --git a/docs/images/tools/per_frame_metric.png b/docs/images/tools/per_frame_metric.png
new file mode 100644
index 0000000..0675806
Binary files /dev/null and b/docs/images/tools/per_frame_metric.png differ
diff --git a/docs/images/tools/per_frame_plotter.png b/docs/images/tools/per_frame_plotter.png
new file mode 100644
index 0000000..84d3b1a
Binary files /dev/null and b/docs/images/tools/per_frame_plotter.png differ
diff --git a/docs/images/tools/scores_plotter.png b/docs/images/tools/scores_plotter.png
new file mode 100644
index 0000000..b1dc748
Binary files /dev/null and b/docs/images/tools/scores_plotter.png differ
diff --git a/scripts/deep_analysis_plotter.ipynb b/scripts/deep_analysis_plotter.ipynb
new file mode 100644
index 0000000..be19865
--- /dev/null
+++ b/scripts/deep_analysis_plotter.ipynb
@@ -0,0 +1,169 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "aadf8bdc-2b8a-4737-992f-0ebf7e66cd8f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Install dependencies into the active kernel\n",
+ "%pip install flatten_json pandas matplotlib"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbdc923a-44fd-4250-990b-80771446bbf0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "import json\n",
+ "from collections import defaultdict\n",
+ "from flatten_json import flatten\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pathlib"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "59b6a968-ad86-48a0-8948-d1f42bf0c3c6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "files = [\n",
+ " 'a_deep_analysis.json',\n",
+ " 'b_deep_analysis.json',\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a60a4c63-b884-4558-98bd-ee3d14bbc569",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def aggregate_loops_passes(loops):\n",
+ " results_per_frame = []\n",
+ " num_loops = len(loops)\n",
+ " for loop_results in loops:\n",
+ " for frame_index, frame_results in enumerate(loop_results[\"per_frame_results\"]):\n",
+ " if frame_index >= len(results_per_frame):\n",
+ " results_per_frame.append(defaultdict(int))\n",
+ " results_per_frame[frame_index]['sequence_time_ns'] = frame_results['sequence_time_ns']\n",
+ " for command_buffer_timings in frame_results[\"command_buffer_timings\"].values():\n",
+ " for scope_name, scope_timings in command_buffer_timings[\"scope_timings\"].items():\n",
+ " # A pass can run multiple times per frame; sum its durations, average over loops.\n",
+ " for scope_timing in scope_timings:\n",
+ " results_per_frame[frame_index][scope_name] += (\n",
+ " scope_timing[\"end\"] - scope_timing[\"start\"]\n",
+ " ) / num_loops / 1_000_000 # in ms\n",
+ " if frame_results[\"metrics\"] is not None:\n",
+ " for metric_name, metric in frame_results[\"metrics\"].items():\n",
+ " # TODO: Flatten this in rust to fan_speed_rpm\n",
+ " if metric is not None and metric_name != \"timestamp\":\n",
+ " results_per_frame[frame_index][metric_name] += metric / num_loops\n",
+ " # TODO: Aggregate CPU timings\n",
+ " return pd.DataFrame([flatten(x) for x in results_per_frame])\n",
+ "\n",
+ "\n",
+ "# Load every input file, aggregating its loops/passes into one number per pass per frame.\n",
+ "results = {}\n",
+ "for path in files:\n",
+ " with open(path, \"r\") as json_file:\n",
+ " results[path] = aggregate_loops_passes(json.load(json_file))\n",
+ "\n",
+ "# Concat into one frame: (input file, frame) per row, metric per column\n",
+ "full_dataset = pd.concat(results)\n",
+ "full_dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7de3e2b9-0470-40a2-9ed7-c2d931aaf153",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "# Print all possible metrics\n",
+ "full_dataset.columns.tolist()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6046a029-f421-4a9e-a9d6-9889488ff874",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "metrics = full_dataset\n",
+ "\n",
+ "# Reshape into sequence time + metric type per row, input file per column\n",
+ "metrics = metrics.reset_index().set_index(['sequence_time_ns', 'level_0']).drop('level_1', axis=1)\n",
+ "metrics = metrics.stack().unstack(1).reset_index()\n",
+ "\n",
+ "# From ns to s\n",
+ "metrics['sequence_time_s'] = metrics['sequence_time_ns'] / 1_000_000_000\n",
+ "metrics = metrics.drop('sequence_time_ns', axis=1)\n",
+ "\n",
+ "# Drop the leftover index name so it doesn't show up as a \"level_0\" legend title\n",
+ "metrics.columns.name = None\n",
+ "metrics"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e6740136-668a-44a6-9e35-53f80d586ea5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pathlib.Path('output_analysis').mkdir(parents=True, exist_ok=True)\n",
+ "\n",
+ "# One image per metric/pass: compares all input files over the benchmark timeline.\n",
+ "for graph_name in metrics['level_1'].unique():\n",
+ " # Grab just this metric, leaving one column per input file plus the timeline\n",
+ " selected_metric = metrics[metrics['level_1'] == graph_name].drop('level_1', axis=1)\n",
+ "\n",
+ " ax = selected_metric.infer_objects(copy=False).interpolate(method='linear').plot(\n",
+ " x='sequence_time_s',\n",
+ " xlabel='benchmark timeline in seconds',\n",
+ " ylabel='shader execution time in ms',\n",
+ " title=graph_name,\n",
+ " figsize=(20, 10),\n",
+ " colormap='Dark2',\n",
+ " grid=True,\n",
+ " legend=True,\n",
+ " )\n",
+ " ax.figure.savefig(f'output_analysis/{graph_name}.png')\n",
+ " plt.close(ax.figure) # free the figure so we don't keep ~94 open at once\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.12.10"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scripts/frame_breakdown_stackplot.ipynb b/scripts/frame_breakdown_stackplot.ipynb
new file mode 100644
index 0000000..2ccc7ff
--- /dev/null
+++ b/scripts/frame_breakdown_stackplot.ipynb
@@ -0,0 +1,194 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "aadf8bdc-2b8a-4737-992f-0ebf7e66cd8f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Install dependencies into the active kernel\n",
+ "%pip install flatten_json pandas matplotlib"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bbdc923a-44fd-4250-990b-80771446bbf0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "import json\n",
+ "from collections import defaultdict\n",
+ "from flatten_json import flatten\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pathlib"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "59b6a968-ad86-48a0-8948-d1f42bf0c3c6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "files = [\n",
+ " 'a_deep_analysis.json',\n",
+ " 'b_deep_analysis.json',\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a60a4c63-b884-4558-98bd-ee3d14bbc569",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def aggregate_loops_passes(loops):\n",
+ " results_per_frame = []\n",
+ " num_loops = len(loops)\n",
+ " for loop_results in loops:\n",
+ " for frame_index, frame_results in enumerate(loop_results[\"per_frame_results\"]):\n",
+ " if frame_index >= len(results_per_frame):\n",
+ " results_per_frame.append(defaultdict(int))\n",
+ " results_per_frame[frame_index]['sequence_time_ns'] = frame_results['sequence_time_ns']\n",
+ " for command_buffer_timings in frame_results[\"command_buffer_timings\"].values():\n",
+ " for scope_name, scope_timings in command_buffer_timings[\"scope_timings\"].items():\n",
+ " # A pass can run multiple times per frame; sum its durations, average over loops.\n",
+ " for scope_timing in scope_timings:\n",
+ " results_per_frame[frame_index][scope_name] += (\n",
+ " scope_timing[\"end\"] - scope_timing[\"start\"]\n",
+ " ) / num_loops / 1_000_000 # in ms\n",
+ " if frame_results[\"metrics\"] is not None:\n",
+ " for metric_name, metric in frame_results[\"metrics\"].items():\n",
+ " # TODO: Flatten this in rust to fan_speed_rpm\n",
+ " if metric is not None and metric_name != \"timestamp\":\n",
+ " results_per_frame[frame_index][metric_name] += metric / num_loops\n",
+ " # TODO: Aggregate CPU timings\n",
+ " return pd.DataFrame([flatten(x) for x in results_per_frame])\n",
+ "\n",
+ "\n",
+ "# Load every input file, aggregating its loops/passes into one number per pass per frame.\n",
+ "results = {}\n",
+ "for path in files:\n",
+ " with open(path, \"r\") as json_file:\n",
+ " results[path] = aggregate_loops_passes(json.load(json_file))\n",
+ "\n",
+ "# Concat into one frame: (input file, frame) per row, metric per column\n",
+ "full_dataset = pd.concat(results)\n",
+ "full_dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7de3e2b9-0470-40a2-9ed7-c2d931aaf153",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Print all possible metrics\n",
+ "full_dataset.columns.tolist()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6046a029-f421-4a9e-a9d6-9889488ff874",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "metrics = full_dataset\n",
+ "\n",
+ "# Reshape into sequence time + metric type per row, input file per column\n",
+ "metrics = metrics.reset_index().set_index(['sequence_time_ns', 'level_0']).drop('level_1', axis=1)\n",
+ "metrics = metrics.stack().unstack(1).reset_index()\n",
+ "\n",
+ "# From ns to s\n",
+ "metrics['sequence_time_s'] = metrics['sequence_time_ns'] / 1_000_000_000\n",
+ "metrics = metrics.drop('sequence_time_ns', axis=1)\n",
+ "metrics"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e6740136-668a-44a6-9e35-53f80d586ea5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pathlib.Path('output_analysis').mkdir(parents=True, exist_ok=True)\n",
+ "\n",
+ "# Keep the N biggest passes individually and roll everything else into \"other\",\n",
+ "# so the stackplot stays readable instead of drowning in ~94 tiny bands.\n",
+ "TOP_N = 20\n",
+ "\n",
+ "# Plot all metrics stacked on top of each other, one stackplot per input file,\n",
+ "# so we get a nice overview of how the total time is composed over the benchmark.\n",
+ "for file_name in files:\n",
+ " file_data = full_dataset.loc[file_name]\n",
+ "\n",
+ " # x axis: benchmark timeline in seconds\n",
+ " x = file_data['sequence_time_ns'] / 1_000_000_000\n",
+ "\n",
+ " # All metric columns (everything except the timeline), interpolated and gap-filled\n",
+ " metric_cols = [c for c in file_data.columns if c != 'sequence_time_ns']\n",
+ " stack_data = (\n",
+ " file_data[metric_cols]\n",
+ " .infer_objects(copy=False)\n",
+ " .interpolate(method='linear')\n",
+ " .fillna(0)\n",
+ " )\n",
+ "\n",
+ " # Order by mean contribution, keep the top N, bucket the rest into \"other\"\n",
+ " order = stack_data.mean().sort_values(ascending=False).index.tolist()\n",
+ " top, rest = order[:TOP_N], order[TOP_N:]\n",
+ " plot_data = stack_data[top].copy()\n",
+ " if rest:\n",
+ " plot_data[f'other ({len(rest)} passes)'] = stack_data[rest].sum(axis=1)\n",
+ "\n",
+ " # Biggest contributors at the bottom, \"other\" on top; distinct color per band\n",
+ " labels = list(plot_data.columns)\n",
+ " colors = plt.get_cmap('tab20')(range(len(labels)))\n",
+ "\n",
+ " fig, ax = plt.subplots(figsize=(20, 10))\n",
+ " ax.stackplot(x, plot_data.T.values, labels=labels, colors=colors)\n",
+ " ax.set_title(f'{file_name} - all metrics stacked (top {TOP_N} + other)')\n",
+ " ax.set_xlabel('benchmark timeline in seconds')\n",
+ " ax.set_ylabel('shader execution time in ms')\n",
+ " ax.set_xlim(x.min(), x.max())\n",
+ " ax.margins(y=0)\n",
+ " ax.grid(True)\n",
+ "\n",
+ " # Reverse the legend so its order matches the top-to-bottom order of the stack\n",
+ " handles, leg_labels = ax.get_legend_handles_labels()\n",
+ " ax.legend(handles[::-1], leg_labels[::-1], loc='upper left', ncol=2, fontsize='small')\n",
+ "\n",
+ " fig.tight_layout()\n",
+ " fig.savefig(f'output_analysis/{file_name}_stackplot.png')\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.12.10"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scripts/per_frame_plotter.ipynb b/scripts/per_frame_plotter.ipynb
new file mode 100644
index 0000000..e2484bc
--- /dev/null
+++ b/scripts/per_frame_plotter.ipynb
@@ -0,0 +1,137 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1d9d9451-25fe-40d7-a67e-2fcaf6a9f54a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Install dependencies into the active kernel\n",
+ "%pip install pandas matplotlib"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4a5f8126-43bc-4cba-b65b-136cbf8971ab",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pathlib\n",
+ "pd.set_option('display.float_format', lambda x: '%.3f' % x)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "959dce8e-4f96-4505-ba35-0300c3aee0a1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "files = [\n",
+ " 'a_deep_analysis.json',\n",
+ " 'b_deep_analysis.json',\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a5f9ea81-7bd2-4f70-a0b8-45164afe8f76",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = {}\n",
+ "for file in files:\n",
+ " file_scores = pd.read_csv(file)\n",
+ " # Drop all-zero (empty) columns\n",
+ " file_scores = file_scores.loc[:, (file_scores != 0).any(axis=0)]\n",
+ " scores[file] = file_scores\n",
+ "scores = pd.concat(scores)\n",
+ "\n",
+ "# Frame time = delta between consecutive sequence timestamps\n",
+ "scores['Frame time (ms)'] = (scores['Sequence Time (ns)'].shift(-1) - scores['Sequence Time (ns)']) / 1_000_000\n",
+ "# The final frame's delta is negative (the benchmark loops) and the first 3 frames\n",
+ "# are huge outliers, so drop both.\n",
+ "scores['Frame time (ms)'] = scores['Frame time (ms)'].mask(\n",
+ " (scores['Frame time (ms)'] < 0) | (scores['Frame'] < 3)\n",
+ ")\n",
+ "\n",
+ "# Reshape into row per input file + sequence time, column per metric\n",
+ "scores = scores.reset_index().set_index(['level_0', 'Sequence Time (ns)']).drop(['level_1', 'Loop Index', 'Frame'], axis=1)\n",
+ "\n",
+ "# Convert the nanosecond timing columns to milliseconds\n",
+ "for col in ['Ray Tracing', 'Acceleration Structure Build', 'Rasterization', 'Compute', 'Driver']:\n",
+ " scores[f'{col} (ms)'] = scores.pop(f'{col} (ns)') / 1_000_000\n",
+ "scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b4bbc8ad-fbf4-4c96-8aa4-afb02ff305a0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Reshape into row per sequence time + metric, column per input file\n",
+ "new_scores = scores.stack().unstack(0).reset_index()\n",
+ "new_scores.columns.name = None # drop the leftover index name so it isn't a \"level_0\" legend title\n",
+ "\n",
+ "# ns to s for the sequence time\n",
+ "new_scores['Sequence Time (s)'] = new_scores['Sequence Time (ns)'] / 1_000_000_000\n",
+ "new_scores = new_scores.drop('Sequence Time (ns)', axis=1)\n",
+ "new_scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2d6b6f75-251b-4bb2-aa44-e6393cf8140a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pathlib.Path('output_per_frame').mkdir(parents=True, exist_ok=True)\n",
+ "for graph_name in new_scores['level_1'].unique():\n",
+ " graph_data = new_scores[new_scores['level_1'] == graph_name].drop('level_1', axis=1)\n",
+ " graph_data = graph_data.apply(pd.to_numeric, errors='coerce').interpolate(method='linear')\n",
+ "\n",
+ " ax = graph_data.plot(\n",
+ " x='Sequence Time (s)',\n",
+ " xlabel='benchmark timeline in seconds',\n",
+ " ylabel='value',\n",
+ " title=graph_name,\n",
+ " figsize=(20, 5),\n",
+ " colormap='Dark2',\n",
+ " grid=True,\n",
+ " rot=0,\n",
+ " )\n",
+ " ax.figure.savefig(f'output_per_frame/{graph_name}.png')\n",
+ " plt.close(ax.figure) # free the figure so we don't keep one open per metric\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python [conda env:base] *",
+ "language": "python",
+ "name": "conda-base-py"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.13.5"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/scripts/scores_plotter.ipynb b/scripts/scores_plotter.ipynb
new file mode 100644
index 0000000..5d670dc
--- /dev/null
+++ b/scripts/scores_plotter.ipynb
@@ -0,0 +1,103 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c7a3c3bd-54d5-4979-8502-ac69a32a13fb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Install dependencies into the active kernel\n",
+ "%pip install pandas matplotlib"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "21658370-184d-4ab0-b210-d8ab93f0818b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "import os\n",
+ "import matplotlib.pyplot as plt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "07bb93e8-5d29-4bbf-8f44-be5db0af70c7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "files = [\n",
+ " 'a_deep_analysis.json',\n",
+ " 'b_deep_analysis.json',\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9088c71c-0fe8-4c87-852e-265b662dcacc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "scores = {}\n",
+ "\n",
+ "for file in files:\n",
+ " # Read csv\n",
+ " file_scores = pd.read_csv(file)\n",
+ " # Remove empty results\n",
+ " file_scores = file_scores.loc[:, (file_scores != 0).any(axis=0)]\n",
+ " # Use os.path.basename() to get only the file name\n",
+ " file_name = os.path.splitext(os.path.basename(file))[0]\n",
+ " scores[file_name] = file_scores\n",
+ "# Reshape into row per metric, column per input file\n",
+ "scores = pd.concat(scores).groupby(level=0, sort=False).mean().T.drop('Loop', errors=\"ignore\")\n",
+ "scores"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d89bab16-7553-4fd2-953b-05a054362999",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "os.makedirs('output_scores', exist_ok=True)\n",
+ "ax = scores.plot(\n",
+ " kind=\"bar\",\n",
+ " figsize=(20, 5),\n",
+ " colormap='Dark2',\n",
+ " grid=True,\n",
+ " rot=0,\n",
+ " title=\"Evolve scores (higher is better)\",\n",
+ " ylabel=\"score\",\n",
+ ")\n",
+ "ax.figure.savefig('output_scores/evolve_scores.png', bbox_inches='tight')"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.13.2"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}