feat: Evaluation Module Prototype

[frayle-ons]

✨ Summary

These changes add a new evaluation module that can quantitatively evaluate the performance of VectorStores against a user-provided ground truth dataset using package-provided classification metrics.

The current implementation views the evaluation as a 'multi-class-single-label' problem, with each evaluation metric assessing only the top candidate result output from the VectorStore search method. Additional metrics could be considered such as 'hit@k' where we assess if the correct label is in the top K results of a VectorStore search, to give users more options on how to evaluate VectorStores.

The main entry point for the new code; the evaluation function, importable from the Evaluation module, has several parameters:

• one or more VectorStore objects or callable functions that would instantiate a vectorstore (allowing users to efficiently instantiate each VectorStore only when its evaluation run starts, saving memory),
• a list of string names to keep track of each corresponding vectorstore,
• a ground truth dataframe of ['qid', 'text', 'label'], where text is the query, and label is the ground truth correct label,
• a list of metrics to calculate (currenctly `['acccuracy', 'macro_precision', 'macro_recall', and 'macro_f1'] are available,
• (optionally) an output file name to save the results to.

The evaluation function performs several actions:

1. checks the validity of the input arguments
2. searches each vectorstore with the provided eval data queries.
3. computes specified metrics for each vectorstore search results using provided ground truth labels
4. collects all metric results for all vectorstores
5. returns the dataframe of results with metric names as columns and row index as the provided name of the vectorstore. Saves dataframe to file if output_file argument is provided with a valid .csv string.

Final thought:

• Currently the ground truth data assumes there is 1 label. We could make this module accept different kinds of ground truth data: one gold standard label per sample, or multiple ground truths per sample. This could mean we have different evaluation modes where the internal logic is different based on which kind of dataframe is passed (and possibly we have a separate argument in the evaluate function that specifies what mode of evaluation we're doing.

📜 Changes Introduced

• (feat:) introduces new evaluation function
• (feat:) new error classes and pandera schemas to ensure data
• (feat:) metrics.py to collect metrics to evaluate.
• (feat:) init file for new package module
• (feat:) smart vectorstore loading for memory efficiency

✅ Checklist

Passes all pre-commit checks.

🔍 How to Test

The tester should obtain some data that has queries and gold standard labels for a give vectorstore, loading the queries into a pandas dataframe with ['qid', 'text', 'label'] columns. All columns should be loaded with string types.

Next they should instantiate the corresponding VectorStore(s) they wish to evaluate against the ground truth dataset.

Passing these items are arguments to the new evaluate function as in the code below, and specifying a list of metric names, the user can run the script to see the evaluation function,

from classifai.evaluation import evaluate 


results = evaluate(
    vectorstores = [vectorstore1, vectorstore2],
    vectorstore_names = ["vectorstore_1", "vectorstore_2"],
    metrics = ["macro_precision", "accuracy"],
    ground_truth = eval_data_df,
    output_file="./classifai_eval_results.csv"
)

Additionally test the functionality that allows users to pass callable to the vectorstores argument of the eval function. To do this the user could write a zero-arg function that returns an instantiated vectorstore object and pass the function pointer as an argument (not the call the to function). See below for the normal process of instantiation versus the same vectorstore wrapped as a callable.

from classifai.vectorisers import HuggingFaceVectoriser
from classify.indexers import VectorStore


# normal instantiation 
vectoriser = HuggginFaceVectoriser('model_name'='sentence-transformers/all-MiniLM-L6-v2')

# normal vectorstore
vectorstore = VectorStore(
    file_name="./testdata.csv",
    data_type="csv",
    vectoriser=vectoriser,
)


# building a function that returns a vectorstore only when its time for its evaluation run.
def second_vectorstore():

    built_vectoriser = HuggingFaceVectoriser(model_name="sentence-transformers/all-MiniLM-L6-v2")

    built_vectorstore = VectorStore(
        file_name="./testdata.csv",
        data_type="csv",
        vectoriser=built_vectoriser,
        )
    
    return built_vectorstore


# this time passing the pointer to the callable to the second item in the list
results = evaluate(
    vectorstores = [vectorstore, second_vectorstore],
    vectorstore_names = ["in_memory_vectorstore", "memory_efficient_vectorstore"],
    metrics = ["macro_precision", "accuracy", "macro_recall", "macro_f1"],
    ground_truth = eval_data_df,
)

Finally try out some edge cases such as:

• Passing items that aren't a vectorstore or callable, or a callable that does not return a vectorstore object.
• make vectorstore_names and vectorstores different lengths.
• try passing a metric that does not exist
• pass the classification_suite string to the metrics, which triggers all current metrics to run
• pass a ground_truth dataframe with missing columns

[lukeroantreeONS]

This is a great looking feature, looking forward to being able to use it.

I've added a few comments / change requests to make this initial version easier to build on in future, but overall it's fantastic and a nice piece of work.

[lukeroantreeONS]

As discussed, I think we can drop qid as a required column for the user to provide, and can optionally export with the dataframe index referenced if it's useful to indicate particular rows to the user at some point.

[lukeroantreeONS]

Up to you whether to remove, construct from dataframe index, or just export with the index directly.

[lukeroantreeONS]

I think this module could be more flexible, and could be extended more easily without breaking changes, if instead of these metrics being functions they were objects inheriting from a base class that has an
.evaluate() method returning a Metric object with (initially) a .value attribute.

This would provide scope to develop the Metric object further at a later date to (for example) add a .coverage attribute for % of unique KB labels included in the evaluation dataset, .confidence attribute to indicate statistical significance of reported metric value, etc.

[lukeroantreeONS]

As you clarified in the PR body, this initial version will be limited to multi-class, single label evaluation. However, a probable requirement for later iterations will be to have (optional) multi-class, multiple-label evaluation, so ideally our implementation for this initial version will lay some groundwork towards that to minimise breaking changes in future.

E.g. if we were to change this to label (str | List[str]), with a note that the option to provide a list is not yet in use and if a list is passed only the first element will be considered.

Alternatively - we could mark this evaluation module with a warning on import, and throughout our documentation for the first version, that this module should be considered as a 'preview' feature, and breaking changes within it will not be restricted to major version updates.

I'm happy with either of these options, let me know what your preference would be.

[lukeroantreeONS]

(question, not change request)

Would it be more flexible if we were to have the functionality in this file exist within some kind of Evaluation class (not fussy on the name if you prefer another), which allows some configuration to be controlled by the user?

e.g.

my_eval = Evaluation(
    ground_truth=some_df,
    batch_size=100,
    save_output=False
)

I would picture a possible workflow with this approach looking like;

eval_sic = Evaluation(
    ground_truth=sic_df,
    batch_size=100,
    save_output=False
)
eval_soc = Evaluation(
    ground_truth=soc_df,
    batch_size=100,
    save_output=False
)

eval_sic.evaluate( # required to generate metric objects as class attributes for use by other methods
    vector_stores=...,
    metrics=...,
)
eval_sic.generate_report(
    output_file=...,
    report_misclassifications=False,
    ...
)
eval_sic.visualise(
    style=...,
    savefile=False
)
...