Performance Evalution¶

This toolbox provides a BaseEvaluator class for evaluating recognition performance. Users can use this class as the father class to write your own evaluator or use the above given functions or classes to write your own evaluation process.

The BaseEvaluator class is a trial based evaluator. Evaluator contains several evaluation trials and evaluate performance trial by trial. Each trial contains several training and testing trials. In each trial, the BaseEvaluator uses the given training trials to train all models one by one and then tests their performance in testing trials. The training time, evaluation time, ture labels and predicted labels will be stored. The recognition accuracies and ITRs can be further computed.

Evaluator¶

SSVEPAnalysisToolbox.evaluator.BaseEvaluator()¶

Initialize the evaluator.

Parameters

dataset_container – A list of datasets. Each element is a instance of one dataset class introduced in “Datasets”.
model_container – A list of recognition models/methods. Each element is a instance of one recognition model/method class introduced in “Recognition algorithms”.

trial_container –

A list of trials. The format is

[[train_trial_info, test_trial_info],
 [train_trial_info, test_trial_info],
 ...,
 [train_trial_info, test_trial_info]]

where train_trial_info and test_trial_info are instances of the TrialInfo class.

save_model –
If True, trained models in all trials will be stored in trained_model_container. The format of trained_model_container is
```
[[trained_model_method_1, trained_model_method_2, ...],
 [trained_model_method_1, trained_model_method_2, ...],
 ...,
 [trained_model_method_1, trained_model_method_2, ...]]
```
where trained_model_method_1, trained_model_method_2, … are instances of recognition model/method classes, which order is same as model_container.

If False, trained_model_container is an empty list.

Default is False.
disp_processbar – If True, a progress bar will be shown in console to illustrate the evaluation process. Otherwise, the progress bar will be shown. Default is True.
ignore_stim_phase – If True, stimulus phases of generating reference signals will be set as 0 during the evalution. Otherwise, stimulus phases will use the dataset information. Default is False.

Note

Saving models by setting save_model as True may occupy large memory.

BaseEvaluator.save()¶

Save the evaluator. If the whole evaluator cannot be saved, only performance_container will be saved.

Parameters: file – Saved path.

BaseEvaluator.load()¶

Load existed evaluator. If the whole evaluator cannot be loaded, only performance_container will be reloaded.

Parameters: file – Local file path of the existed evalutor.

BaseEvaluator.run()¶

Run the evaluation process. Performance will be stored in performance_container. The format of performance_container is

[[performance_method_1, performance_method_2, ...],
 [performance_method_1, performance_method_2, ...],
 ...,
 [performance_method_1, performance_method_2, ...]]

where performance_method_1, performance_method_2, … are instances of the PerformanceContainer class for different recognition models/methods. The order follows model_container.

Parameters

n_jobs – Number of threadings using for recognition methods. If the given value is larger than 1, the parallel computation will be applied to improve the computational speed. Default is None, which means the parallel computation will not be applied. The evaluator will reset n_jobs in recognition methods as None. Parallel computation will be applied to evaluation trials.
eval_train – Please ignore this parameter and leave this parameter as the default value. The function related to this parameter is under development.

Trial Information¶

SSVEPAnalysisToolbox.evaluator.TrialInfo()¶

The instances of this class are the basic elements of trial_container in BaseEvaluator.

It contains following attributes:

dataset_idx: A list of dataset indeices.
sub_idx: A list of all datasets’ subject index list. The format is
```
[[sub_idx_1, sub_idx_2, ...],
 [sub_idx_1, sub_idx_2, ...],
 ...,
 [sub_idx_1, sub_idx_2, ...]]
```
where sub_idx_1, sub_idx_2, … are subject indices for different datasets. The order follows dataset_idx.
block_idx: A list of all datasets’ block index list. The format is same as sub_idx but the integer numbers in lists are block indices.
trial_idx: A list of all datasets’ trial index list. The format is same as sub_idx but the integer numbers in lists are trial indices.
ch_idx: A list of all datasets’ channel index list. The format is same as sub_idx but the integer numbers in lists are channel indices.
harmonic_num: The harmonic number is used to generate reference signals. One integer number.
tw: The signal length (in second). One float number.
t_latency: A list of latency times of datasets. Each element is a float number denoting a latency time of one dataset.
shuffle: A list of shuffle flag. Each element is a bool value denoting whether shuffle trials.

TrialInfo.add_dataset()¶

Push one dataset information into the trial information

Parameters

dataset_idx – dataset index. One integer number.
sub_idx – List of subject indices. A list of integer numbers.
block_idx – List of block indices. A list of integer numbers.
trial_idx – List of trial indices. A list of integer numbers.
ch_idx – List of channel indices. A list of integer numbers.
harmonic_num – The harmonic number is used to generate reference signals. This input parameter will update harmonic_num of the trial information. One integer number.
tw – The signal length (in second). This input parameter will update tw of the trial information. One float number.
t_latency – Latency time (in second). A float number.
shuffle – If True, the order of trials will be shuffled.

Returns

The instance itself.

get_data()¶

Based on the trial information, get all data, labels, and reference signals.

Parameters

dataset_container – List of datasets.

Returns

X: List of all EEG trials.
Y: List of all labels.
ref_sig: This function will use the first dataset in dataset_idx to generate reference signals.
freqs: List of stimulus frequencies corresponding to generated reference signals.

Note

This TrialInfo will only use the first dataset to generate reference signals. If datasets have different stimuli, please separate them into different trials. The more safety way is that one TrialInfo cotains only one dataset.

Performance Container¶

SSVEPAnalysisToolbox.evaluator.PerformanceContainer()¶

The instances of this class are the element of performance_container in BaseEvaluator.

It contains following attributes:

true_label_train: After training, to evaluate the training performance, the list of true labels of training trials is stored in this attribute. The format is
```
[[true_label_1, true_label_2, ...],
 [true_label_1, true_label_2, ...],
 ...,
 [true_label_1, true_label_2, ...]]
```
where true_label_1, true_label_2, … are true labels of different evaluation trials.
pred_label_train: After training, to evaluate the training performance, the list of predicted labels of training trials is stored in this attribute. The format is same as true_label_train.
true_label_test: The list of true labels of testing trials is stored in this attribute. The format is same as true_label_train.
pred_label_test: The list of predicted labels of testing trials is stored in this attribute. The format is same as true_label_train.
train_time: A list of storing time of training the model. Each element in the list is one training time of one evaluation trial.
test_time_train: A list of storing time of using the training trials to testing the model. Each element in the list is one testing time of one evaluation trial.
test_time_test: A list of storing time of using the testing trials to test the model. Each element in the list is one testing time of one evaluation trial.

Supplementary functions¶

SSVEPAnalysisToolbox.evaluator.gen_trials_onedataset_individual_diffsiglen()¶

Generate trial_container for BaseEvaluator. These evaluation trials only use one dataset. One block is used for testing. Other blocks for training. All blocks will be tested one by one. All subjects will be evaluated one by one for each signal length.

Parameters

dataset_idx – Dataset index. One integer number.
tw_seq – List of signal lengths (in second). A list of float numbers.
dataset_container – List of datasets.
harmonic_num – Number of harmonics. One integer number.
trials – List of trial indices. A list of integer numbers.
ch_used – List of channel indices. A list of integer numbers.
t_latency – Latency time (in second). A float number. If None, the suggested latency time of the dataset will be used.
shuffle – If True, trials will be shuffled. Default is False.

SSVEPAnalysisToolbox.evaluator.cal_performance_onedataset_individual_diffsiglen()¶

Calculate evaluation performance of BaseEvaluator whose trial_container is generated by gen_trials_onedataset_individual_diffsiglen.

Parameters

evaluator – The instance of the class BaseEvaluator.
dataset_idx – Dataset index.
tw_seq – List of signal lengths (in second)
train_or_test – If "train", evaluate performance of training trials. If "test", evaluate performance of testing trials.

Returns

acc: Classification accuracy. The shape is (methods × subjects × signal length).
itr: ITR. The shape is (methods × subjects × signal length).

SSVEPAnalysisToolbox.evaluator.cal_confusionmatrix_onedataset_individual_diffsiglen()¶

Calculate confusion matrices of BaseEvaluator whose trial_container is generated by gen_trials_onedataset_individual_diffsiglen.

Parameters

evaluator – The instance of the class BaseEvaluator.
dataset_idx – Dataset index.
tw_seq – List of signal lengths (in second)
train_or_test – If "train", evaluate confusion matrices of training trials. If "test", evaluate confusion matrices of testing trials.

Returns

confusion_matrix: Confusion matrices. The shape is (methods × subjects × signal lengths × true classes × predicted classes).

Plot Functions¶

SSVEPAnalysisToolbox.evaluator.shadowline_plot()¶

Plot shadow lines. Each group plots one shadow line.

Parameters

X – List of variable values.
Y – Plot data. The shape is (groups × observations × variables). The line is the mean across observations. The shadow is the variation across observations.
fmt – Format of lines. Default is '-'.
x_label – Label of x axis. Default is None.
y_label – Label of y axis. Default is None.
x_ticks – X tick labels. Default is None.
legend – List of line names. Default is None.
errorbar_type – If 'std', calculate the variation using the standard derivation. If '95ci', calculate the variation using the 95% confidence interval.
grid – Whether grid. Default is True.
xlim – [min_x, max_x]. Default is None.
ylim – [min_y, max_y]. Default is None.
figsize – Figure size. Default is [6.4, 4.8].

SSVEPAnalysisToolbox.evaluator.bar_plot_with_errorbar()¶

Plot bars with error bars. Each group plots one color bars.

Parameters

Y – Plot data. The shape is (groups × observations × variables). The bar height is the mean across observations. The error bar is the variation across observations.
bar_sep – Separate distence of adjacent bars.
x_label – Label of x axis. Default is None.
y_label – Label of y axis. Default is None.
x_ticks – X tick labels. Default is None.
legend – List of bar names. Default is None.
errorbar_type – If 'std', calculate the variation using the standard derivation. If '95ci', calculate the variation using the 95% confidence interval.
grid – Whether grid. Default is True.
xlim – [min_x, max_x]. Default is None.
ylim – [min_y, max_y]. Default is None.
figsize – Figure size. Default is [6.4, 4.8].

SSVEPAnalysisToolbox.evaluator.bar_plot()¶

This function is similar as bar_plot_with_errorbar. But this function only plots one group data and does not plot error bars.

Parameters

Y – Plot data. The shape is (observations × variables). The bar height is the mean across observations. The error bar is the variation across observations.
bar_sep – Separate distence of adjacent bars.
x_label – Label of x axis. Default is None.
y_label – Label of y axis. Default is None.
x_ticks – X tick labels. Default is None.
grid – Whether grid. Default is True.
xlim – [min_x, max_x]. Default is None.
ylim – [min_y, max_y]. Default is None.
figsize – Figure size. Default is [6.4, 4.8].