[Feature]:SequenceAccuracy Metric for PyTorch Ignite

🚀 Feature

I think the addition of a SequenceAccuracy metric (or an enhancement to the existing Accuracy metric) to natively support sequence-to-sequence outputs common in NLP and Transformer-based models will be good.

Motivation:

When working with sequence models, the model output is typically of shape (batch, sequence_length, num_classes) and the target is (batch, sequence_length). Currently, the standard ignite.metrics.Accuracy expects inputs to be already flattened or simple 1D/2D tensors.

I have to write a custom output_transform for every single project to flatten the tensors and create masking.

Current work around:

def get_metrics(loss_fn, pad_idx=0):
    def accuracy_transform(output):
        preds, y = output

        preds_flat = preds.reshape(-1, preds.size(-1))
        y_flat = y.reshape(-1)

        mask = y_flat != pad_idx

        preds_masked = preds_flat[mask]
        y_masked = y_flat[mask]

        return preds_masked, y_masked

    return {
        "accuracy": metrics.Accuracy(output_transform=accuracy_transform),
        "loss": metrics.Loss(loss_fn),
    }

As Accuracy is one of the most common nlp metric I think offering a more plug and play option will be godd.

We can create a new metric SequenceAccuracy which can be a wrapper around Accuracy and perform masking, alternatively we can modify the Accuracy itself to support sequences and masking.