NLP | Fine Tuning BERT to perform Spam Classification

In this video, We will show you how to fine-tune a pre-trained BERT model using PyTorch and Transformers library to perform spam classification on a dataset.
Here are some important links -
Article: www.analyticsv....
Github Repo: github.com/pra...

[@hemalshah1410]

Simply crisp explanation of content on surface level . Thank you for making such contribution towards learning Community. Kudos !

[@alidi5616]

Great walkthrough with clear explanation and keeping it simple. Thank you so much. Exactly what i needed. :)

[@ganeshkharad]

this is one of best walkthrough i have seen👍👍

[@Zelloss67]

I do not understand where is a funetuning of the Bert takes place. You freezed gradients in Bert so Bert didnt adjust itself for your specific dataset. For example if you didnt freeze Bert tensors the gradient could overflow to Bert so it generate tokens, which are better representation of text in you dataset.
What I saw just now was you training neural network for multi-class classification, which uses Bert tokens as inputs.
Could you please provide some qlarifications on misunderstanding mentioned above?
Thank you very much

[@Analyticsvidhya]

You're absolutely right! Fine-tuning BERT in this case involves using its pre-trained embeddings and lower layers as a fixed feature extractor for your specific spam classification task. Freezing these layers doesn't mean BERT doesn't learn anything.
Here's why:
➡️ The "fixed" BERT layers still represent complex relationships between words and sentences, providing valuable context for your task.
➡️ The newly added neural network (your "BERT_Arch") is where the actual fine-tuning happens. This network learns to interpret and weigh the information extracted by BERT to distinguish spam from non-spam emails.
➡️ Freezing BERT prevents gradient overflow issues that could arise from backpropagating through its massive pre-trained parameters. This allows for stable and efficient training of your specific classification head.
In short, you're training a new classifier on top of a powerful pre-trained feature extractor, where the "fine-tuning" happens in the newly added layers, not directly in the pre-trained BERT weights.
Remember, freezing BERT is a common and effective technique in fine-tuning, especially when dealing with limited datasets. ❤️

[@vrbabu228]

Nice video! What should be changed if we were to look into multi class text classification? I know that the output layer dimension needs to be changed. Anything else to be changed?

[@kaverianuranjana9787]

Even I would be interested in a multiclass classification implementation (maybe NLI?) that is as clear as this example. Thanks!

[@urbanholds4944]

Where is format_time function while evaluating the model?

[@kakamondol1322]

In the bert architecture if I want to concatenate some extra static feature before fc layer , how to do?

[@victorthomas6844]

Can you please suggest to me how to use the Bert model to compare two sentences for semantic similarity and assign them to a class. Your suggestions would be appreciated. Thanks.

[@itsme1674]

Very nice explanation

[@avirajbevli7268]

Nice tutorial, but this is not "Fine tuning BERT" in the true sense since you are freezing the weights in the BERT model. You are essentially doing "Feature Extraction", i.e. using the feature vectors obtained from the BERT model and giving that as an input to our simple classification model(The 2 linear layers defined in the model) and only training the final 2 linear layers to perform spam classification.

[@basmahhyder5695]

Could you elaborate what do you mean by not freezing the BERT model parameters?

[@archanadurgam3711]

i'm getting an error in Start Model Training
NameError Traceback (most recent call last)
in ()
7
8 #for each epoch
----> 9 for epoch in range(epochs):
10
11 print('
Epoch {:} / {:}'.format(epoch + 1, epochs))
NameError: name 'epochs' is not defined
help me with this please

[@carlopeluso974]

lol

[@tahahuraibb5833]

can we add a data loader for the prediction? I keep getting cuda out of memory error.

[@kumarvaibhav7203]

The "Start Training Model" is not working when I tried code from github. This gives attribute error
#train model
train_loss, _ = train()
AttributeError: 'str' object has no attribute 'dim'

[@manarhamad782]

Yes , i got the same problem

[@kumarvaibhav7203]

@@manarhamad782 I found the fix. They are using Tensorflow 3.0. Just add this line as first line - !pip install transformers==3.0.0

[@manarhamad782]

Vaibhav Kumar ok I will try ,, thank you very much

[@vikassalaria24]

@Analytics Vidya After fixing this, when training the model getting NameError: name 'cross_entropy' is not defined

[@ammaarahmad5747]

@@vikassalaria24 How did you fix this error?

[@sanketjaiswal6687]

how to visualize this above model in pytorch?

[@hasanbasriakcay]

Thank you for video. May I learn Transformers version and Tensorflow version?

[@dw8200]

Nice walkthrough. Thanks for sharing.

[@lukasnielsen1263]

Very nice tutorial, clear architecture and nice and structured code. But you are not really fine-tuning a BERT model (as you freeze all the BERT models parameters), but rather the two dense layers atop the BERT model.
It's important to underscore this, as the BERT model parameters remained unchanged after your fine-tuning, thus the BERT model is not fine-tuned, the classification "head" you added is :)

[@lukasnielsen1263]

​@fsociety Yes, if you dont freeze the BERT parameters, they are fine-tuned. An easy way to check if this is the case is to encode the same word before and after fine-tuning the bert model, and compare the before and after vectors.

[@basmahhyder5695]

Could you elaborate what do you mean by not freezing the BERT model parameters?

[@lukasnielsen1263]

@@basmahhyder5695 By freezing the parameters, they are not updated during fine-tuning. If you dont freeze them (which is standard practise btw) then the BERT model parameters are updated during fine-tuning

[@alexsudakov8554]

class_wts = compute_class_weight('balanced', np.unique(train_labels), train_labels)
TypeError: compute_class_weight() takes 1 positional argument but 3 were given
How to fix?

[@alexsudakov8554]

Fixed!
class_wts = compute_class_weight(class_weight = 'balanced', classes = np.unique(train_labels), y = train_labels)

[@НикитаНагорный-ч3о]

Nice work!!!

[@amenmechlaoui8717]

bro i need data of call center fraude , where can i find this data ?

[@Analyticsvidhya]

Try searching on Kaggle

[@jimharrington2087]

Nice coding walkthrough!!

[@sachinborse4178]

Please make on video as soon as possible because there is error showing in start the training train_loss,_=train() i try my best to solve this but no any single trick working🙏🏻

[@Analyticsvidhya]

Let me get back to you on this..

[@anbinh3967]

unreal! thanks!

[@tonyafields4432]

THANK YOU!

[@khushbootaneja6739]

Nice

[@Analyticsvidhya]

Thanks

[@andrea-mj9ce]

I can't manage to read the data (1:05)

[@Analyticsvidhya]

We'll suggest you to set video to 1080p. Then, your issue should be resolved.

[@andrea-mj9ce]

@@Analyticsvidhya I was meaning to read the data in my computer with `read_csv`

[@moshiwei8454]

would be nice to reduce the accent a bit

[@HariNarayan-s7t]

I'm getting this error:
Epoch 1 / 10
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in ()
12
13 #train model
---> 14 train_loss, _ = train()
15
16 #evaluate model
4 frames
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py in forward(self, input)
112
113 def forward(self, input: Tensor) -> Tensor:
--> 114 return F.linear(input, self.weight, self.bias)
115
116 def extra_repr(self) -> str:
TypeError: linear(): argument 'input' (position 1) must be Tensor, not str
How do I fix this ?

[@Analyticsvidhya]

It looks like you're encountering a TypeError in your code. The error message indicates that there's an issue with the input data type. Specifically, the linear function is expecting a PyTorch Tensor as input, but it's receiving a string (str) instead.
To fix this issue, check the part of your code where you're passing data to the linear function and ensure that you're passing a PyTorch tensor with the correct shape and data type.

[@kartikeylohani6934]

@@Analyticsvidhya Hi, I am getting the same error, when i am printing the cls_hs variable, it is giving the output as pooler_output.
PS please respond fast

[@tianxiaoye-gn7jf]

@@Analyticsvidhya Hi, I met the same error, I think there may be some changes in the bert, making the output becomes str.

[@tianxiaoye-gn7jf]

In the block """class BERT_Arch(nn.Module):'''''''
#pass the inputs to the model
_, cls_hs = self.bert(sent_id, attention_mask=mask, return_dict=False)
Add the third arg, then it works

[@Analyticsvidhya]

Thank you for sharing your experience and solution

[@RinoNucara]

I get an error, solved:
from sklearn.utils.class_weight import compute_class_weight
#compute the class weights
class_wts = compute_class_weight('balanced', classes=np.unique(train_labels), y=train_labels)
print(class_wts)

[@Analyticsvidhya]

Hey Rino, so your error is solved, right?

[@RinoNucara]

@@Analyticsvidhya yes! Thank you!