Right now, backends are decided by a string that is internally matched with a set of accepted strings:
benchmark = Benchmark(
name="visualwebarena",
backends=["visualwebarena"],
# ...
)
It would be good if we can have a class called Backend that must have a method called prepare, which is called when the backend is being prepared. This would allow anyone to submit a custom backend without having to rewrite the Benchmark class, e.g. backends=[MyBackend()]
Right now, backends are decided by a string that is internally matched with a set of accepted strings:
It would be good if we can have a class called
Backendthat must have a method calledprepare, which is called when the backend is being prepared. This would allow anyone to submit a custom backend without having to rewrite theBenchmarkclass, e.g.backends=[MyBackend()]