Transfer Data from S3 bucket to Snowflake via pandas | Complete Code & Installation from Scratch

A Silly Mistake in function definition.
Please Forgive me for that.
In the Video Connection is given as input parameter but conn was used inside the function .
Just Corrected it below.
Dataset:
github.com/Sat...
Installing the Python Connector
docs.snowflake...
Using Pandas DataFrames with the Python Connector
docs.snowflake...
Python Code:
import pandas as pd
import snowflake.connector as sf
import boto3
from snowflake.connector.pandas_tools import write_pandas
service_name='s3'
region_name='us-east-2'
aws_access_key_id=''
aws_secret_access_key=''
s3 = boto3.resource(
service_name=service_name,
region_name=region_name,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key
)
user=""
password=""
account="";
database="RAMU"
warehouse="COMPUTE_WH"
schema="PUBLIC"
role="SYSADMIN"
conn=sf.connect(user=user,password=password,account=account);
def run_query(conn, query):
cursor = conn.cursor();
cursor.execute(query);
cursor.close();
statement_1='use warehouse '+warehouse;
#statement2='alter warehouse '+warehouse+" resume";
statement3="use database "+database;
statement4="use role "+role;
run_query(conn,statement_1)
#run_query(conn,statement2)
run_query(conn,statement3)
run_query(conn,statement4)
for obj in s3.Bucket('demoshow').objects.all():
df=pd.read_csv(obj.get()['Body'])
df.columns = ['SEPALLENGTH', 'SEPALWIDTH', 'PETALLENGTH', 'PETALWIDTH', 'CLASSNAME'];
write_pandas(conn, df, 'VIDEO')
print(df)
Snowflake Code:
create database if not exists ramu;
use ramu;
--Table Creation
create table if not exists video (sepallength number(10,4),sepalwidth number(10,4),petallength number(10,4) , petalwidth number(10,4),classname varchar(200));
select * from video;
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY TH-cam CHANNEL

[@mirzauzair6854]

Keep up the good work! Great video and explanation.

[@KnowledgeAmplifier1]

Glad you liked it Mirza Uzair! Happy Learning :-)

[@prateekgurani5982]

Thanks a lot! Great way of explanation.

[@KnowledgeAmplifier1]

Glad it was helpful Prateek Gurani! Happy Learning :-)

[@adarshsurendran4347]

​ @KnowledgeAmplifier1 Amazing and rare content bro..subscribed! ... Could you tell me , here, you are executing the python code from the IDE... in production, where would you put this python code and how would you run/schedule it?

[@KnowledgeAmplifier1]

Hello Adarsh Surendran, Thank you for your support ! And regarding the question , in production, the code will run in some system (may be EC2 , EMR , Lambda , Glue or some on-prem cluster e.t.c.) and it can be scheduled using Autosys , Airflow or cloudwatch etc.
To explore more , you can refer my recent videos on scheduling ETL Pipelines using Airflow :th-cam.com/video/oH-O7rrwnOg/w-d-xo.html
Hope this will be helpful! Happy Learning

[@adarshsurendran4347]

@@KnowledgeAmplifier1 Thank you for replying. Sure, I will check out this video.

[@chundurusriharsha2402]

If I have .bak file in S3 bucket Then how can I do it?

[@cleversachin]

good job.. but at the end .. i though like watching a cricket thriller.. Keep it up