Biomedical Named Entity Recognition with Transformers

This is a tutorial on how to annotate and recognize biomedical entities with the Bio-Epidemiology-NER package and the biomedical-ner-all model.
Bio-Epidemiology-NER is a Python library built on top of the biomedical-ner-all model to recognize bio-medical entities from a corpus or a medical report.
Research Paper: journals.plos.org/digitalheal...
Authors: Shaina Raza, Deepak John Reji, Femi Shajan, Syed Raza Bashir
Package: pypi.org/project/Bio-Epidemio...
GitHub: github.com/dreji18/Bio-Epidem...
Huggingface Hub: huggingface.co/d4data/biomedi...
This package can recognize over 50 different entity types, including clinical entities (disease, symptoms, risks, effects, drugs, diabetes, respiration, vital signs, and others), as well as non-clinical entities, such as event-based data, social factors that are not clinical factors but are related to health outcomes. Second, with no code changes, this pipeline is simple to use and adaptable to individual methods for a given data type, task, or domain of application. Third, this pipeline can take any free texts, for example, in the form of text or PDF files and parse them for scientific texts. We hope that this package will provide a more transparent and customizable solution for the healthcare industry, helping to educate and encourage more rigorous applications of ML to biomedical analyses.
Chapters
00:00 Introduction
00:24 About the model (biomedical-ner-all)
02:19 About the package (Bio-Epidemiology-NER)
03:30 how to use the model
06:09 how to use the package
09:04 Report annotation feature
15:36 Conclusion

[@kitanomegumi1402]

This is a good model that I've been using for my course project for some time. Your work is very much appreciated!

[@deepakjohnreji]

Thank you so much :)

[@user-vk7wg2up3o]

Its not working. ner_prediction(corpus=doc, compute='cpu')
AttributeError: 'DataFrame' object has no attribute 'append'

[@caiyu538]

great lectures, hugging face is a great platform to provide so many excellent AI models to us. Thumb up your great lecture

[@deepakjohnreji]

Thank you so much :)

[@user-xd7op6it9n]

Great work. Kindly provide the training notebook, codes on how to train the model. Thanks in advance

[@shivasj]

Great work Deepak! Have you published the notebook for training for this work someplace yet?

[@deepakjohnreji]

Hi Shivas, thanks. I haven't published the training notebook yet; I will notify here once its published :)

[@shivasj]

@@deepakjohnreji Thanks! looking forward to it

[@ZakariyaFirachine]

hey ,can you please provide the training notebook .thanks in advance

[@deepakjohnreji]

Hi, the notebook is not being shared, the research paper has its details for training the model

[@hayarmen2807]

Good afternoon! Tell me, please, have you published a file with the training of the model? I really like your work and I want to develop in this field!

[@deepakjohnreji]

Hi, Thank you for watching, I haven't published the code files yet, the research paper has the details of the model: journals.plos.org/digitalhealth/article?id=10.1371%2Fjournal.pdig.0000152

[@caiyu538]

Where can I use this to train my customer data? I still use hugging face trainer to train it? I have my own classifications and may not need so many categories for classification. Thank for great lectures.

[@deepakjohnreji]

If you are using the trainer api from huggingface, try loading this model, during the model and tokenizer loading step and use your data and categories.

[@caiyu538]

@@deepakjohnreji Thank you for your confirmation. I think I should continue to use hugging face trainer to train it. If I use my own category ( only have 3-4 categories), would it conflict with the category of your model (you have 84 categories). How to solve this potential issue (I have not done it yet, maybe this is not an issue at all). I will check my boss if our categories are within your categories. I would prefer to use your model because your model is intensively trained using medical report data and it is fine tuned. Thank you so much. Thumb up your great lecture again.

[@manfyegoh]

Interesting! any chance you will publish codes/tutorial on how to train the model? thank you very much

[@deepakjohnreji]

Thanks, would be doing it soon !!!

[@manfyegoh]

@@deepakjohnreji Thank you very much, I love your content alots! keep it up!

[@deepakjohnreji]

@@manfyegoh Thank you for your kind words :)

[@thexjoker5022]

@@deepakjohnreji im sorry for my rudeness, but if you will make the tutorial on how to train the model, when you will release it?, thanks for your efforts!

[@abhiksarkar3859]

Very useful work. But i am getting AttributeError: 'DataFrame' object has no attribute 'append'. can you pls recheck/update the code?

[@snehashajipunnan2904]

how can it be solved?

[@andrearomeroramos6586]

I don't understand the difference between biomedical-ner-all model and the distilBERT model

[@caiyu538]

This model focus on using medical report data to train. Great to use it for medical data. This is my understanding.

[@deepakjohnreji]

Hi, so this model is a finetuned version on biomedical data.

[@andrearomeroramos6586]

@@deepakjohnreji Yes, but "biomedical-ner-all" is only de name of the model? the finetuned model (using distilbert?