Thanks! With Rasa, if you work with our new dialogue system CALM, you can use intents (classify with an NLU), go intentless (use an LLM to understand incoming utterances in context), or both simultaneously. That means some rasa projects still leverage intents, making everything in this video still very relevant. Annotation remains important even if you take an intentless approach and use a language model to categorize user utterances. While you might not use it to collect intent examples, annotation can still be very useful to evaluate how well your system is working to quantify system performance and simply to evaluate the user experience by reading real user conversations and annotating them to see what is working well or not working. Hope that helps!
(Vincent here) Could you clarify? One general concern in ML is that your training data does not reflect what your users are interested in. Since the world is a moving target, that means that you should keep labelling new data that's coming in just to ensure you're able to recognise what folks are asking for. This might also mean that you need to update your test set as new data from the changing world comes in.
@@RasaHQ Thank you for your reply! What I understood from the video is that we use annotations by hand. That made me wonder what will we do for new data we did not yet see and did not annotate. The annotations, as I understood them- are a type of features - and if they do not exist for the test data they are not useful, or did I miss something?
If you're multilingual, a lot of translation/linguist jobs involve a lot of data annotation. Unfortunately a lot of annotating is done through crowdsourcing, which often doesn't pay well enough to be a full time job. - R
Your videos are great but there's no code and does not seem to be aimed for developers either, more like a general understanding of each component that a person without coding experience would perfectly understand. I'm a devolper and was expecting some code examples on each of the videos of this series but there's none
This series is designed as a high level intro to help developers with requirements and design decisions, rather than a tutorial on the coding portion. (And for most of the videos there is either no single canonical approach or, like for this one, no coding is necessary). You can check out our tutorials or live coding sessions if you're looking for code specifically. - Rachael
"Common errors and gotchas: Not doing data annotation." - You do not know, and you will never really know, how frickin' TRUE this statement is!
Nice video: How has this annotation landscape changed with advent of LLMs?
Thanks!
With Rasa, if you work with our new dialogue system CALM, you can use intents (classify with an NLU), go intentless (use an LLM to understand incoming utterances in context), or both simultaneously. That means some rasa projects still leverage intents, making everything in this video still very relevant.
Annotation remains important even if you take an intentless approach and use a language model to categorize user utterances. While you might not use it to collect intent examples, annotation can still be very useful to evaluate how well your system is working to quantify system performance and simply to evaluate the user experience by reading real user conversations and annotating them to see what is working well or not working.
Hope that helps!
Great video. Thanks for the info and resources at the end!
Always perfect and up to date ❤️
I want to annotate data for aspect based sentiment analysis. Can you pls help me in that
Thank you for the video!
I have a question: How is annotating any good for unseen test data?
(Vincent here)
Could you clarify? One general concern in ML is that your training data does not reflect what your users are interested in. Since the world is a moving target, that means that you should keep labelling new data that's coming in just to ensure you're able to recognise what folks are asking for.
This might also mean that you need to update your test set as new data from the changing world comes in.
@@RasaHQ
Thank you for your reply!
What I understood from the video is that we use annotations by hand.
That made me wonder what will we do for new data we did not yet see and did not annotate.
The annotations, as I understood them- are a type of features - and if they do not exist for the test data they are not useful, or did I miss something?
i really interested with data annotation, anyone know the training and job for this?
If you're multilingual, a lot of translation/linguist jobs involve a lot of data annotation. Unfortunately a lot of annotating is done through crowdsourcing, which often doesn't pay well enough to be a full time job. - R
Thanks a lot!
please tell me how can take free practice test of data annotation? before going to interview of data annotation job
I'd recommend Zooniverse: www.zooniverse.org/ --Rachael
about "Expert-annotated"
meaning find a human expert in the field to handle it manually?
Very good information, I wish I've had seen this before starting my project 😂
At least now you know for next time! :)
Your videos are great but there's no code and does not seem to be aimed for developers either, more like a general understanding of each component that a person without coding experience would perfectly understand. I'm a devolper and was expecting some code examples on each of the videos of this series but there's none
This series is designed as a high level intro to help developers with requirements and design decisions, rather than a tutorial on the coding portion. (And for most of the videos there is either no single canonical approach or, like for this one, no coding is necessary). You can check out our tutorials or live coding sessions if you're looking for code specifically. - Rachael