Loss Function — Recurrent Neural Networks(RNNs)

In the last article, we discussed the Data Modeling jar for Sequence labeling and sequence classification problems. In this article, we touch upon the loss function the sequence-based problems

Loss Function

Sequence Classification Task

Let’s say there are “2 classes positive and negative” and in this case (snippet below), the “actual label happens to be positive” which means the “entire probability mass is on the positive class/label” (true distribution)

Now “y_hat” would be computed using the model equation for Recurrent Neural Networks (RNNs)

And let’s assume that the model predicts the following distribution for this case:

As it’s a classification problem and there are two probability distributions, the Cross-Entropy Loss is used to compute the loss value:

Please note that it would be ‘y_hat’ instead of ‘y’ in the second line — it denotes the negative logarithm value of the predicted value corresponding to the true class

Cross entropy loss value — it is a “negative summation” over “all possible values that random variable can take” (in this case the possible values are 0 and 1), and the “product of the true probability” and the “logarithm value of the predicted probability”