Training a model to recognize sentiment in text (NLP Zero to Hero - Part 3)
ฝัง
- เผยแพร่เมื่อ 2 มี.ค. 2020
- Welcome to Zero to Hero for Natural Language Processing using TensorFlow! If you’re not an expert on AI or ML, don’t worry -- we’re taking the concepts of NLP and teaching them from first principles with our host Laurence Moroney (@lmoroney).
In the last couple of episodes you saw how to tokenize text into numeric values and how to use tools in TensorFlow to regularize and pad that text. Now that we’ve gotten the preprocessing out of the way, we can next look at how to build a classifier to recognize sentiment in text.
Colab → goo.gle/tfw-sarcembed
GitHub → goo.gle/2PH90ea
NLP Zero to Hero playlist → goo.gle/nlp-z2h
Subscribe to the TensorFlow channel → goo.gle/TensorFlow - วิทยาศาสตร์และเทคโนโลยี
Great, clear, short and right to the point explanation. It is also very helpful to learn the concept more in depth when you give a link to the code that we can practice it by our own. Thank you Laurence
SeqtoSeq and Transformers from scratch please! You're my hero, Laurence!
You have this amazing gift to simplify such difficult concepts.
Tons of love from Myanmar!
The JSON file has to be fixed as follows:
import json
with open('./Sarcasm_Headlines_Dataset.json','r') as f:
lines = f.readlines()
s = '['
for line in lines:
#each line already had a '
' so I had to strip it out, add a comma, and put the '
' back in
s+= line[:-1] +',
'
#the last line will have an extra ',
' which we need to get rid of and cap with a ']'
s= s[:-2]+']'
with open('./sarcasm_fixed.json', 'w') as f:
f.write(s)
with open('sarcasm_fixed.json', 'r') as f:
datastore = json.load(f)
you are incredible. I'm looking a solution to this problem for hours in net. thank you so much
you save the day my friend, thanks
Thanks, man. Kudos!
Time series please? You’re my favorite professor!!
Your Explanation is Mindblowing Sir ! Your are the real NLP Hero :)
Amazing content! The shackles of complex NLP jargon has now been broken. You are doing a HUGE favor to soo many people (me included) out there by sharing NLP essence in such a simple way! Thank you!
Again a great tutorial and demonstration. Thank you very much!
Welcome!
Awsome!! Waiting for this video from the last week
Thanks, Thananjaya! :)
I opened TH-cam expecting to see part 3....and I find it on top.
Thanks @ moroney
Hope you enjoy!
Links were very helpful, you explained everything clearly and simply. Thank you
for those who are looking for ways to fix the json file:
data is equivalien to datastore in the video, which is a list of items (dictionary)
import json
data = []
with open("sarcasm_fixed.json", 'r') as f:
lines = f.readlines()
for i in lines:
print(i) # optional - just print it out to explain the next 2 lines
i.replace("'","") # this is to replace " ' " with ""
i.replace("/n","") # this is to replace "/n" with ""
obj = json.loads(i)
data.append(obj)
thanks broo
can you explain both of the i.replace() lines, what you are doing is trying to change a string, but string in python is immutable and that line basically returns a new modified line which we are not storing so basically its useless, so in order to bring the change, u have to write i = i.replace().
also still not clear why we are doing it, kindly explain, thank you!
This is great, any idea why the input json file kaggle does not follow standard json array syntax such as [{},{}]
This worked for me. Thanks!
Love the short yet succinct explanation of concepts on this channel. PS. @lmoroney you remind me a lot of Philip Seymour Hoffman esp from his role in MI-3!
in a very easy way he gave us very clear understanding wit example , a huge salute
I am loving Tensorflow more than me !!
Great tutorial for people just starting with NLP.. Now anyone can write a usable NLP system.. NLP for the masses 👍
Very awesome series! Thank you!
This is so clear and easy to understand. Thank you, now I have a better understanding about NLP.
prolly the best explanation for embedding layer 🎉
Is this the end of NLP Zero to Hero? I wanna see so much more.
Edit: No, it isn't.
Such a talent! Thanks.
Thanks for your efforts.
awesome series
Great tutorials! Can't wait for part 4!
There's only 3...sorry! :)
You are the man !! 👨🏽🎓🧑🏽💻🤟🏾
My fav professor
I am a professor myself, but my teaching quality is nowhere near yours. You are the best.
Amazing! Keep up the good work!
How'd you get a heart?
GIVE IT TO ME!!! IT'S MINE!!!!!
Short video but rich contents. Thanks
Great! I try to keep them less than 6 minutes, but had to go a bit longer this time
Really thank you
You did cover it well.
Thanks!
what do you think of adding a Conv1D after the embedding layer to extract features from the embeddings? would that help?
thank you for awesome tutorial, ^_^ Waiting for text generating video.
Been delayed, unfortunately, because of events moving virtual in response to the Covid-19 outbreak. But there's lots of great content coming from TF Dev Summit, Cloud.NEXT and more...
Thank you for this tutorial! Question: What is the best way to determine the number of epochs to use?
Can you explain, why you've chosen the embedding_dim to be 16? Is that an arbitrary number you've chosen, or do you base it on a factor from the dataset?
amazing job!
How you decide "Good-Bad": goodness and the other possibilities?
P.D.: the other interesting part for me is that regarding communication the challenge is still up for all the body language, as important or more as the linguistics.
If I'm trying to read more on "common or garden deep neural network" what do I search for? I didn't get immediate hits for the latter
do you have a video tutorial about coreference resolution using tensorflow and keras?
Nice video. Have few doubts
How does it know that a word should be plotted on some coordinates and what is the relation of padded Sequence with embedding process
Hi, I face this error when I tried to load json file : JSONDecodeError: Extra data: line 2 column 1 (char 217)
What is the source of this error?
Thanks
The JSON file is not a valid JSON. Use some editor (such as Notepad++), open the .json file and replace
with ,
(this adds a comma at the end of every line) and then add [ at the beginning of the file and ] and the end of the file, also delete the last comma. This will make it a valid JSON.
how many filters are used in the Sequentail model?? I think we must have filters for pooling to use .
Thanks. Can you make the final two indexes easier to interpret? (Humans
numbers)
Waiting for the next video on this awesome Nlp video series
Thanks, preparing to shoot some "Coding TensorFlow: At Home" episodes while staying home to prevent the spread of Covid-19
How do I decide how great the number of the embedding_dim should be? Why is it 16 in the example?
what algorithm used in this tutorial ?
can we get a series on abstractive text summarization??
Could someone please explain what I should put for the output and input dimensions for the embedding layer?
awesome videos! subscribed. What IDE you used to get such colorful outputs?
It doesn't depend on the IDE, sir.
Very useful
Glad you like!
Love the videos.
Quick doubt I had though, are the tokenizations of words in any way related to their meanings? Where words with similar meanings are close to each other numerically
No, I believe that the tokens are assigned randomly based on the order of the sentences and do not have any relation to the meanings of the words but however I may be wrong.
Nice!!!
Thanks :)
Excellent explanation, especially the one of the "Embedding Layer". I have always struggled to understand it from other videos and articles. Thank you very much.
Just a quick question. So from the explanation, the embedding layer seems to learn the relationship of the words to the dependent variable. In the case of a Recurrent Nueral Network, would it then learn how the features relate to the sequences that are learnt by an RNN? I ask this mainly because I have seen other people putting an embedding layer with an RNN when building a chatbot?
I'm particularly interested in applying an RNN to some time series data, say for stock prices, I believe that making use of extra features like the company issuing the stock etc. then feeding the time series with the extra feature of company name through an embedding layer and then through an RNN might improve the model?
Thank you!
Kinda sorta. Consider the embedding to be a set of numbers that describe the word, including its semantics. RNNs are good at determining meaning in a sequence. So, the sentence "In Ireland I went to high school, so I had to learn how to speak " has the value of determined very early on in the sentence by the word 'Ireland', so maintaining meaning in a long sequence is important, and RNNs (and in particular LSTMs) are good at that.
For the stock sequence thing...I'm not so sure. If the 'external' data can be shown to have an impact on the trend, then possibly. If it doesn't (for example, the name), then I would doubt it.
@@LaurenceMoroney So well explained! Really appreciate it! Looking forward to more Machine Learning tutorials, really learning a lot!
What is the output dimention of the GlobalAveragePooling layer ? (1, )??
how do run these predictions over a csv file : in one column rows of sentences and second columns appended results
Pekase do a short series on Automatic speech recognition or audio enhancement, etc etc
Oh that would be nice. Not sure if it would be short though!
For those wondering, skillingsbolle is a type of norwegian pastry
@Laurence, thank you for the insightful content. is it possible to determine the time spent in training a model from the perspective of planning ML activities? are there any guide lines around this?
Depending on your dataset, the time will vary. For example, if your dataset is a set of audio consisting of total data of 300MBs, the first epoch takes around 10-15 seconds, and thereafter all the epochs take ~1s to complete. So, ideally, your dataset will be in GBs and you would require a lot of training, thousands of epochs, maybe, so expect a couple of hours of training-only phase in your project. Now, if your project is for a fairly general purpose, you may get away with minimal training, ranging from a couple of seconds to an hour, because if you notice, when you create something like a face recognition lock in your phone, it asks for training data by asking you too look at the device, while it captures photos. Now, that takes only about half a minute to train and test and finally store the model, but as you know, still performs in a fairly robust manner. The use-case I described above was for huge production-ready critical application softwares, for example, a suicide-prevention software that may provide Machine-Learned AI Counsellors. There exists a similar platform, as far as I know, and was even featured on the tensorflow channel. Now that had a training time (total, over a period of time) of around 5000 hours, because they could not afford mistakes in their execution, so... completely depends on your use-case buddy don't let anything hinder you. Go on and build something great. By the way, if you haven't heard of it, a website called "Teachable Machine", created by Google helps you train models impressively fast, and it also allows you to export your model to use in your apps, along with all the required stuff (well basically other than the tflite model, only the labels file is required, which it ships to you along the model itself in a zipped format). Now, as long as you do not want something like on-device training, teachable machine is a great way to get started, even in production apps (although I would recommend testing with self-made models before deploying). Good luck,
@@rohanmanchanda5250 thanks for taking time respond and in a very detailed manner. Quick follow up. you mentioned the use case of face recognition, is the model present/updated on the device at all times or takes the help of a remote service ? also how do we account for long training cycles? can this activity be run in parallel by leveraging multiple GPUs? Another question is, do we always need GPUs for training, testing and execution? or for any of phases we can get by by using commodity hardware?
@@praskatti whenever the model is present on-device or is fetched remotely is entirely up to the developer, in the case of system software like the face recognition service, it is up to the OEM. There are a couple of possibilities...
1.) You can store the model on-device and train on-device - simple enough, everything is stored in the apps assets and no network connectivity is required.
2.) You can store the model on-device but train in the cloud - some training procedures may require powerful machines to complete fast and so you may wanna upload the training data to a server and train the model in-cloud.
3.) Model in Cloud and training on-device : again, fairly simple, the model is in the cloud. You upload the training data and train the model in the cloud.
4.) Model on-cloud, training on-device: This is somewhat rare since it's not exactly much useful, but may be preferable if model size is large and you do not wish to bloat the app size with it. However, keep in mind that users will still need to download the model which will cost them data. So, it is usually better to ship the model along with the app itself.
Generally, you do not require on-device training for simple models, like object detection. You can pre-train your model and just apply it to use.
If you do wish to train the model based on the user-specific parameters, consider using on-device training instead of training-in-cloud, since the latter would require uploading the users' data to a server, which many users who value privacy would not appreciate.
Apart from training, the model, if stored in cloud, should be deployed wisely. If you download the model from the cloud, cache it while your app is being used, and then discard it, this will cost the users a lot of data every time the model is downloaded, i.e., every time your app is used. Your app would become data-heavy. On the other hand, if you include the model within the app, this is not a problem, but your model must be reasonably sized.
If your model is heavy and you must store it on a server, consider creating a mechanism for model storage. Like download the model once, and store it in the app's storage. Then re-use it from there every other time.
Most of this would not be something you'll have to deal with since you most of the ml projects are dealing with simple small-sized models which are pre-trained and shipped as is, not requiring further processing.
If you are developing for mobile, make sure you quantize your modles to integer format, it drastically reduces model size, and also dramatically increases the inference speed, with minimal loss in accuracy.
Try out teachable machine, and if you're comfortable with a little coding in python, check out the Tflite Model Maker API. You can create models with just four lines of code with those, using them is super easy, barely an inconvenience.
Not quite sure how the Y component of the bad-good vectors were playing a role. You said the vectors were two dimensional but it seemed that only the X axis made any impact on the sentiment
for the json.load(f), I'm getting TypeError: the JSON object must be str, bytes or bytearray, not TextIOWrapper. What do I do?
Thank you sir plz explain this Vocab Size. is it a dictionary ?
Is it building while training?
I just chose values of my own and will change it to see if the results improve:
vocab_size = 10000
embedding_dim = 16
max_length = 100
trunc_type='post'
padding_type='post'
oov_tok = ""
training_size = 20000
I have a problem when I load my data with the JSON library. When I do json.load(f), I get : "JSONDecodeError: Extra data: line 2 column 1 (char 208)" I've tried to correct my JSON but nothing works. Why ?
Can someone explain what's happening at 6:35 ?
How do you determine vocabulary size? Is it arbitrary figure or there is a way of determining it?
the same here, I still couldn t find an answer
Is there by any chance a Tutorial on how to use this with SVMs. This Series is really good for NLP beginners and I'm currently struggling with applying the Preprocessed Data to classification using SVMs. It should be simple but hasn't quite clicked yet. Appreciate the helpful insights.
I assume you have managed to do it by now..?!
Can anyone stress more on globalaveragepooling2D layer?
I was following in the part 2. But now i'm not sure to understand the relation between the tokenized word and the json with the article, header and sarcasm thing... I don't understand how this can help me detected what do to in a sentence, for example "Play the movie Titanic", "Can you play a movie which will be Titanic". How with that can I isolate the word titanic, same thing for a music title or an application name... I'm kinda lost, is there a way to take private lessons with you ?
help, anyone know what algorithm used in this tutorial ?
while working on your colab example, reaching Epoch 5 makes the val_loss increase exponentially while val_accuracy drops, i tried adding more neurons, drop out layer but nothing worked, any idea why is this happening?
Weird. Did you follow it completely? Did you change any of the data?
@@LaurenceMoroney no I didnt change anything
@@moodiali7324 Is the colab set for GPU ?
Is this language dependant? I guess by the structure this will work for Spanish or German ?
where can i get sentiment.json file ?
Anyone encounter this error as training starts?
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Cast string to float is not supported
@lmoroney small bug in colab notebook plot_graphs(history, "accuracy") gives key error it should be plot_graphs(history, "acc")
Thanks..in 2.0, it's 'accuracy', and I've fixed and updated the colab for this. Please give it a try :)
@@laurencemoroney655 thanks for update
@@Debugerr When you pass the strings 'accuracy' or 'acc', we convert this to
one of `tf.keras.metrics.BinaryAccuracy`,. I saw this when I typed help(model.compile)
Mr. Moroney, do you not think that this is similar to giving a human a couple of dialogues to study from an alien language, and then asking them to categortize it in two? I mean, the task that you wanted to be performed by the model is indeed done well, but apropos of the scenario I just drew, the model (as the human) would not be able to UNDERSTAND the language. For that, we will need to be drawn again to the classical programming driven by the rules, where we would need to instruct the program to perform tasks based on the words detected and categorized by the model. The only advantage here seems that we will not have to write all possible words that will trigger the action.... or so I thought... If we are designing a model for a custom use-case, we would still need to create hundreds of sentences for the model to first train on and then manually train it for results, I believe that does slightly well than just "if(sentence.containsAnyOf("wordA", "wordB", ""....){ performTask() }".
What are your thoughts?
why do we need to split the data into training and testing, can it be all training? then we can test it with real testing sentences which are presented at the end of this video.
python makes it super easy, barely an inconvenience
when i am importing json file it is giving me error (Extra data: line 2 column 1 (char 208)) can you please help me with this??
is there anyone to explain what are these layers and why we use 'relu' as activation function:
tf.keras.layers.GlobalAveragePooling1D(),
tf.keras.layers.Dense(24, activation='relu'),
Global average pooling is effectively summing up the n-dimensional vectors to simplify.
'relu' ignores everything that's
Will this work for multi-class classification? If I have 4 labels.
Add a softmax layer at the end
@@ramasai1475 Yeah, instead of 'sigmoid' 😁
I was working with numbers and i feel that my test data was leaked what to know how to avoid it
Please reply!!!
@@LaurenceMoroney ''?'' what do i understand by it
I'm a little confused on how they assigned vectors to each words sentement. I get the training data are the words themselves, but where is the list of correctly assigned vectors that they used to train the network on sentement?
I understand the concept, it's the code I'm having trouble with
Cool
Thanks!
Can you please add a link to the playlist, thanks!
Entire 'Coding TensorFlow' playlist is here: th-cam.com/play/PLQY2H8rRoyvwLbzbnKJ59NkZvQAW9wLbx.html
I am getting an error using the sarcasm dataset while opening and reading the file. The error is "JSONDecodeError: Extra data: line 2 column 1 (char 217)".
I am getting the same thing
could someone please advise
thanks
Not sure how, but this cleared the error
dataset = []
sentences = []
labels = []
urls = []
for line in open('path/data.json ', 'r'):
dataset.append(json.loads(line))
for item in dataset:
sentences.append(item['headline'])
labels.append(item['is_sarcastic'])
urls.append(item['article_link'])
Hello, get this error!
FileNotFoundError Traceback (most recent call last)
in ()
1 import json
----> 2 with open("sarcasm.json", 'r') as f:
I’ll try to apply this to medical claims 🧑🏽💻
I tested the code with sarcastic sentences but it did not give me good results...
i have the same problem, i tried an lstm NN with 0.86 accuracy at top, but the "granny spider" example sentence results in a prediction near 0.
Entity detection please!
Someday :)
This is keras for babies.
I am a baby.
I love keras.
And most importantly, I love your tutorials!
Is this the end of NLP zero to hero series @laurence moroney
he said there is more on the way. means no.
@@SamuraiDexter It's the end of the classification one. I'm working on a Text Generation one, but that's been delayed because we had to move TFDS and NEXT away from in-person events to digital ones, so getting studio time is hard. Likely will also have changes for IO, which will again constrain my time getting into the studio to film it. So...it's delayed. Sorry!
Technical or billing issue
Is there any alternative way rather than tokenizing the words. I don't think this is the most optimal solution in NLP industry..
It's really up to you. Could do letters, words, sub-words, or if you can slice a sentence into phrases...but when using LSTMs and RNNs, the context of words can be maintained, so for simple scenarios like this one, I like it :)
i still dont get embeddings 😁
I get this error in colab:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
in ()
10 plt.show()
11
---> 12 plot_graphs(history, "accuracy")
13 plot_graphs(history, "loss")
in plot_graphs(history, string)
3
4 def plot_graphs(history, string):
----> 5 plt.plot(history.history[string])
6 plt.plot(history.history['val_'+string])
7 plt.xlabel("Epochs")
KeyError: 'accuracy'
------------------------------------------------------------------------------------------
for this block of codes:
import matplotlib.pyplot as plt
def plot_graphs(history, string):
plt.plot(history.history[string])
plt.plot(history.history['val_'+string])
plt.xlabel("Epochs")
plt.ylabel(string)
plt.legend([string, 'val_'+string])
plt.show()
plot_graphs(history, "accuracy")
plot_graphs(history, "loss")
I've updated...please try again
Embedding meaning in words.
it is to represent each word with a vector
When i try to load the json file I get this error: line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 217)
importing json did not work for me at all, therefore with a little research I found that we can do it using pandas as well.
import pandas as pd
data = pd.read_json("sarcasm.json", lines=True)
headline = data['headline'].to_list()
is_sarcastic = data['is_sarcastic'].to_list()
@@FINTHRTD Got another solution
dataset = []
for line in open('data/sarcasm.json', 'r'):
dataset.append(json.loads(line))
for item in dataset:
print(item['headline'])
@@piotrszostakiewicz7925 json.loads is giving an error
@@FINTHRTD What error? It's working for me
@@piotrszostakiewicz7925 i am getting the same error as before.
isn't the model overfitted? I mean, the accuracy on validation data seems to be noticably lowerer than on training data
I noticed that, too. The validation accuracy peaks around Epoch 2 or 3, then keeps decreasing until the training is over.
*SARCASM SIGN*
ReallY? :P
Comic Book Guy, sarcastically, says, "Oh, a sarcasm detector. Oh, that's a real useful invention." which causes it to overload with sarcastic data and explode.
The very definition of recursion.
Laurence Moroney The medium is the message
@@marshallmcluhan33 And the bottom line is that the bottom line is the bottom line :)
Code is completely messed up in video, people can only guess what code goes where, and it is like 40% shorter than colab posted in video. I still remember promise that this lessons can be easy even if you do not know Python. Instead of writing code, video just shows part of code assuming you will know where to put it. It is not just add code to end, it can go anywhere.
Like lol 😂
Well....that escalated quickly
It did? :)
@@LaurenceMoroney ......to be honest...I did understood after a 2nd repeat
it's almost as if Google owns both Tensorflow and TH-cam. imagine that.
You might be a few years late to the party. TH-cam and Google have been under the same umbrella for like forever.