Fine-Tune Llama3 using Synthetic Data

how to fine tune Llama-3 model in Google Colab in this tutorial using synthetically generated data. In this video chris not only shows you how to fine tune the model but also shows you his lessons learned, such as diversity of data, why the system prompt makes a difference, generalization and fine tuning to a particular format.
You will not only learn how to fine tune a model but also how to generate synthetic data, and learn what works and what doesn't.
Google Colab
colab.research.google.com/dri...
GIthub for Dataset:
github.com/chrishayuk/chuk-da...
HuggingFace for Model
huggingface.co/chrishayuk/lla...
HuggingFace for DataSet
huggingface.co/datasets/chris...

[@Leo-ph7ow]

Great great content! Please, make a local finetune tutorial. Thanks again!

[@chrishayuk]

it's on the list, i promise

[@kenchang3456]

Wow, how fortunate am I?! I was looking for an example of fine-tuning to change the behavior of a model to act like a counter clerk at an auto parts store and I think I have found it and it's synthetic too. THANK YOU VERY MUCH!

[@JonathanDeCollibus]

chris, fantastic video. i've been looking for this exact answer.

[@chrishayuk]

Super glad this was useful, this vid is a little more raw than normal as my purposely pointing out the errors in the dataset rather than fixing them, but I think it’s useful to understand

[@suryat8848]

clean, and crisp!
brilliant video chris :)
PS: Can you please update the tokenizer part of the code, it's a bit confusing, thanks!

[@tomekatomek5694]

3:00 - Show how to do it on a local machine please

[@chrishayuk]

yes, i need to do that video. i've been distracted by building a faster pipeline for the finetune

[@JonathanDeCollibus]

subscribed.

[@chrishayuk]

Awesome, thank you

[@AT-mx3hn]

I like to guess accents. What is your accent?! There is a obvious primary Scottish element but there are also strong hints of American and weaker hints of possibly English and/or Australian... did you move around a lot or are you just trying to sound more American so TH-cam can understand you better?!

[@chrishayuk]

i'm like a fine wine with lots of elements of different accents. i'm a scot that lives in england that used to live in ireland, spent a lot of time in india, us and travels a lot.

[@AT-mx3hn]

Amazing, thanks for taking the time to answer!

[@Forwardknowlege]

can I fine tune Llama-3 by meta as well ? example >>> meta-llama/Meta-Llama-3-8B-Instruct

[@chrishayuk]

ummm, that is llama-3, is there something specific you're trying to do?

[@felipeekeziarosa4270]

@@chrishayuk non-english legislation would be interesting