Data Engineering Interview: Replicating 2nd round of

Data Engineering Mock Interview
It replicates a typical second-round data engineering interview that occurs in most companies. You can directly watch this video to understand how a second-round interview is typically conducted in the Data Engineering industry.
This exciting new video presents a mock interview with a skilled and experienced Data Engineer. Through insightful questions, we explore the techniques, tools, and technologies that the interviewee has employed in their work, gaining valuable insights into the complex and ever-evolving world of data engineering.
From designing and implementing scalable, high-performance batch processing architectures to working with cutting-edge data processing frameworks like #snowflakes, #apachespark, Airflow, AWS etc.
Our expert guest interviewer Nisha shared her hard-won knowledge and expertise, offering valuable advice and insights for aspiring data engineers and seasoned professionals.
Whether you're just starting out in your career or looking to take your skills to the next level, this interview is an essential resource for anyone interested in the fascinating world of real-time data processing and engineering. So don't miss out - tune in now and discover the secrets of success in this dynamic and exciting field!
🔅 To book a Mock interview - topmate.io/ankur_ranjan/15155
𝗝𝗼𝗶𝗻 𝗺𝗲 𝗼𝗻 𝗦𝗼𝗰𝗶𝗮𝗹 𝗠𝗲𝗱𝗶𝗮:
🔅 Topmate - (Book 1:1 or other sessions)
topmate.io/ankur_ranjan
🔅 LinkedIn - / thebigdatashow
🔅 Instagram - / ranjan_anku
Nisha Bansal (Interviewer) 's LinkedIn profile
🔅 / nisha-bansal-17a97b128
Manoj T (Interviewee)'s LinkedIn profile
🔅 / manoj-t-engineer
#interview #dataengineering #bigdata #apachespark #careerswitch #job #mockinterview

[@niraj835]

with cte as (select user_id,session_start,session_end ,session_type,rank() over(partition by user_id order by session_start )as rnk from event)
select user_id, count(*) as total_streamer_count from event where session_type='streamer' group by user_id having user_id in
(select user_id from cte where rnk=1 and session_type='viewer')

[@syedadnan4910]

with session as:
select userid,session_type,row_num() over(partitionby session_type order by session_start) as number from event
group by userid,session type
order by session_start
select userid,count(session_type)as number of session from session
where session_type=="viewer" and number==1;
I think this should work for the last one

[@hoangminhninh9133]

in the pyspark question, if i use createOrReplaceTempView then use SQL command to get the availableFlag, will you accept the answer. With complex logics, that's how we use SQL, for Python part, we use udf python for the transform productId column... Use right tool for right thing

[@chidanandanayak9609]

please mention mock interview for how many years of experiences in title, i'm switching from web-developing to data engineering, when i watched this it literally gave me anxiety , then i learned its interview for a person with 6 years of experiences

Great Content!! , Appreciate your efforts

[@TheBigDataShow]

Thanks a lot 😊

[@arkodaysen261]

Can you please let me know how to read a json file with duplicate keys in spark with its native api or rdds

[@manojt7012]

ideally in that case you should be getting ambiguous column error. I don't think Spark has any inbuilt capabilities to remove such. Rather you can define an UDF in which you parse your json key and remove the duplicate one

[@arkodaysen261]

@@manojt7012 yes that's what,but the interviewer asked if this can be resolved using spark native APIs or using rdd or dataset..

[@kashrex.]

Where to practise writing pyspark codes? Can you please share any such platform?

[@soumyagupta9301]

Google Colab

[@codingseekho001]

SELECT user_id, number_of_event
FROM (
SELECT session_id, user_id, session_type,
ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY session_start) AS user_session_rank,
COUNT(*) OVER (PARTITION BY user_id) AS number_of_event
FROM event
) stat
WHERE user_session_rank = 1 AND session_type = 'viewer';

[@madhav7811]

Please do some interviews on gcp data engineer as well, at present gcp is booming right so it might helpful for few people's like me.

[@TheBigDataShow]

We are planning this. Stay tuned.

[@avinash7003]

Round 1 link please

[@TheBigDataShow]

Hi Avinash
It replicates a typical second-round data engineering interview that occurs in most companies. You can directly watch this video to understand how a second-round interview is typically conducted in the Data Engineering industry.

[@avinash7003]

@@TheBigDataShow can you make video on present Data engineering market expectations

[@siddharthsingh5031]

Why the interviewer is dead from inside? Anyway great questions.

[@TheBigDataShow]

What do you mean by this? Not everyone is very expressive by face and it does not determine their intelligence. Some are brilliant introverts with not much facial expression & it should be respected Man.

[@siddharthsingh5031]

@@TheBigDataShow I am sorry if it came that way. I meant it as a joke. I resonate with what you said about her intelligence. Anyway, good job bringing these helpful and nice people on this platform.

[@marksun6420]

What is the problem asked by the interviewer on 31:15?

[@TheBigDataShow]

Small File problem

[@user-gl6lu5lp5g]

Not getting what he is saying........i think he is mug up all things just like he is saying

[@TheBigDataShow]

Hey Man
Cool, don't worry 😅 You have to work hard for more than 6 years to reach that level of him. So shows some gratitude to his experience and knowledge

[@user-gl6lu5lp5g]

what she is saying ..what he is answering....in data engineer position she is asking on aws devops.

[@TheBigDataShow]

When you grow in this field for more than 6-8 years then you are expected to build a full fledged product and team at good product based companies. Knowing your infrastructure, deployment strategy, scalability and your expertise at cloud is very much required. This interview is only meant for good product based companies.

[@HarjeetSingh-ne7km]

they really dont ask DE questions, their round one feels like SQL interview

[@TheBigDataShow]

@@HarjeetSingh-ne7km Have you really seen this video. Where is the SQL in it & above that every good product based companies will always ask some SQL and DSA in first round of any Data Engineering interview. So if you are only preparing for service based then you can skip these video. SQL and DSA is truth in first round and one can't run from there.

[@user-gl6lu5lp5g]

Dont take this like interviews other will goes in commmma......😆😆😆😆

[@TheBigDataShow]

If you really want to learn and work in a better place then you can't deny the hard part of the interview. Interviewer and interviewee both are having more than 6 years of experience

[@views-re2om]

i am paralyzed

[@sumeetsaonerkar9322]

God this is too much technical...

[@prabhatgupta6415]

right i got a panic atttack lol

[@TheBigDataShow]

Hey Don't worry 😅
Just prepare with consistency, you will do it.

[@user-gl6lu5lp5g]

pyspark ---zero knowledge.....

[@TheBigDataShow]

Start giving mock interviews online then you will realise how much courage ones required & anyone can get stuck at any problem. It doesn't mean that given a chance he will not be able to do it.

[@rohitmali1587]

​@@TheBigDataShowwhere I can give mock interview online??

[@TheBigDataShow]

Rohit, you message me on LinkedIn with your attached resume. Mention that you want to give mock interview