- 79
- 136 384
Steve Tarzia
United States
เข้าร่วมเมื่อ 10 เม.ย. 2013
CS-310 Lecture 19 - Distributed Computing
Outline:
0:12 Last time - Computing Platforms
2:00 Easy vs. Hard scalability problems
4:10 “Big Data” problems
5:53 Distributed computing
7:22 Example: Find the largest sales order
9:29 Maximum aggregator is trivially parallelizable
13:35 Basic features of distributed computing
14:16 Maximum aggregator is trivially parallelizable
14:17 Basic features of distributed computing
16:07 Hadoop is a distributed computing platform
19:45 MapReduce
21:29 MapReduce programming model
23:17 MapReduce example: counting words
26:54 Counting words example
29:31 Communication between many nodes
30:32 The Hadoop framework provides:
31:49 Spark improves on MapReduce/Hadoop ideas
33:15 Spark innovations
34:25 Resilient Distributed Datasets (RDDs)
37:09 Spark example: Text Search
39:35 Example: Logistic Regression (ML classification)
42:15 Traditional Parallel Computing
44:23 Recap
45:57 Architectural concepts in this class:
46:14 Class Recap
These lectures were delivered by Professor Steve Tarzia in Winter 2021 for his CS-310 Scalable Software Architectures course at Northwestern University. This class was formerly numbered CS-396. Course materials and slides can be downloaded from stevetarzia.com/teaching
0:12 Last time - Computing Platforms
2:00 Easy vs. Hard scalability problems
4:10 “Big Data” problems
5:53 Distributed computing
7:22 Example: Find the largest sales order
9:29 Maximum aggregator is trivially parallelizable
13:35 Basic features of distributed computing
14:16 Maximum aggregator is trivially parallelizable
14:17 Basic features of distributed computing
16:07 Hadoop is a distributed computing platform
19:45 MapReduce
21:29 MapReduce programming model
23:17 MapReduce example: counting words
26:54 Counting words example
29:31 Communication between many nodes
30:32 The Hadoop framework provides:
31:49 Spark improves on MapReduce/Hadoop ideas
33:15 Spark innovations
34:25 Resilient Distributed Datasets (RDDs)
37:09 Spark example: Text Search
39:35 Example: Logistic Regression (ML classification)
42:15 Traditional Parallel Computing
44:23 Recap
45:57 Architectural concepts in this class:
46:14 Class Recap
These lectures were delivered by Professor Steve Tarzia in Winter 2021 for his CS-310 Scalable Software Architectures course at Northwestern University. This class was formerly numbered CS-396. Course materials and slides can be downloaded from stevetarzia.com/teaching
มุมมอง: 1 058
วีดีโอ
CS-310 Lecture 18 - Computing Platforms
มุมมอง 8323 ปีที่แล้ว
Outline: 0:05 Last Time: Twitter Database Architecture 1:24 App packaging, distribution, and deployment is tricky 6:29 Packaging option #0: Binary Executable 10:33 Packaging option #1: Source code README 12:48 Packaging option #2: JVM Bytecode 14:52 Virtual Machines 19:00 Packaging option #2: JVM Bytecode 19:01 Virtual Machines 20:18 Why use virtual machines? 21:05 In the old days, you installe...
CS-310 Lecture 17 - Twitter Design Exercise
มุมมอง 1K3 ปีที่แล้ว
Outline: 0:09 Summarizing the quarter so far! 2:12 Twitter design example 5:05 Original/simplest design 6:45 Why is building my twitter feed slow? 11:05 Second Twitter design 13:21 Why is building my twitter feed slow? 13:22 Second Twitter design 13:35 The slow celebrity tweet 15:45 Our theoretical Twitter architecture 2.0 17:40 Getting your feed 19:27 Review 21:38 Twitter feeds in a NoSQL data...
CS-310 Lecture 16 - Asynchronous Processing
มุมมอง 9623 ปีที่แล้ว
Outline: 0:17 Recap - Choosing a data store 1:39 Ways to couple services 3:48 But what if the request is really long? 5:06 Asynchronous Alternative 7:00 Synchronous to Asynchronous - what changed? 8:14 What if client needs to know the results? 9:18 Option 1: Request Record 11:52 Option 2: Callback to Client 14:02 Option 3: Side channel for feedback 16:20 Requests that would benefit from asynchr...
CS-310 Lecture 15 - Choosing a Database
มุมมอง 1.1K3 ปีที่แล้ว
Outline: 0:21 Last Time: Distributed DB Consistency 2:26 Recall the goals of a Database: 2:53 Data storage options 3:56 SQL Relational Databases 5:50 Database transaction example: spending gift card... 13:50 Transactions on a distributed (NoSQL) DB? 14:43 How to implement a distributed lock? 21:33 Throughput/scaling limitations 22:35 Data abstractions 24:29 Column-oriented Relational Databases ...
CS-310 Lecture 14 - Distributed DB Consistency
มุมมอง 1.2K3 ปีที่แล้ว
Outline: 0:07 Last Time: NoSQL databases 3:52 Hash-based partitioning of distributed DB 4:30 A Shared Nothing architecture 5:29 Making the DHT robust 8:09 Consistency 10:39 CAP Theorem 15:15 Client-centric consistency models 18:51 Client-centric consistency properties 21:23 Failing the Monotonic Read property 26:21 Failing to Read your Writes 27:44 Failing the Monotonic Writes property 31:49 Tw...
CS-310 Lecture 13 - Distributed NoSQL Databases
มุมมอง 1.3K3 ปีที่แล้ว
Outline: 0:06 Last Time: Push Notifications 1:36 Recall from Lecture 9: Scaling SQL Databases 2:26 Review of Sharding: 3:59 Normalized data 5:16 Graph partitioning model for DB sharding 7:02 Normalized data 7:03 Graph partitioning model for DB sharding 8:23 Partitioning challenges 11:34 The jump from SQL to NoSQL 13:20 From Normalized … to Denormalized Data 15:40 NoSQL rationale 17:56 Hashing i...
CS-310 Lecture 12 - Push Notifications
มุมมอง 9203 ปีที่แล้ว
Outline: 0:06 Last Time: Architecture Example 1:34 Limitations of Client-Server Architectures 6:55 Email is a simple way to send notifications 9:11 The Internet is not really a network of peers 16:16 Solution: Push Notification Service 18:44 Smartphone Push Notifications 20:26 The Internet is not really a network of peers 20:44 If you took CS-340: How to deal with NAT? 20:46 Device Registration...
CS-310 Lecture 11 - Basic Architecture Design
มุมมอง 1.3K3 ปีที่แล้ว
Outline: 0:12 Last Time: Authentication 1:43 Case Study: NGVM 4:33 NGVM architecture diagram 13:27 Monolithic web app API: Public Pages 17:22 Monolithic web app API: Volunteers' Portal 20:38 SQL Database Schema (simplified) 24:49 S3 File Store details 28:23 April 2020 monthly operating cost ($136 total) 31:59 CDN statistics in April 33:21 Deployment sizing and monthly costs 37:08 Scaling up to ...
CS-310 Lecture 10 - Authentication
มุมมอง 8533 ปีที่แล้ว
Outline: 0:08 Last time: SQL Database Scaling 2:43 Authentication is proving your identity 6:42 Simplest approach: Password in every request 9:17 Session Keys are like temporary passwords 12:21 Review: Cookies are auth tokens for web browsers 13:20 Every request handler must now check the auth token 15:30 API Keys 16:57 Where to include auth token in a REST API? 18:18 Two factor authentication ...
CS-310 Lecture 09 - SQL Database Scaling
มุมมอง 3.2K3 ปีที่แล้ว
Outline: 0:07 Recap: Storage and Relational Databases 2:15 Memory vs disk access in databases 5:17 Databases are performance bottlenecks 10:46 Relational Database performance optimizations 16:40 Read replicas 20:37 What limits the number of read replicas? 24:31 Multi-level replication can extend read-scalability 29:33 How to use read-replicas? 32:58 Replication shortcomings? 34:27 Primary-prima...
CS-310 Lecture 08 - Relational Databases
มุมมอง 1.4K3 ปีที่แล้ว
Outline: 0:04 Last Time: Load balancers 2:54 On to Databases 4:09 Back to Wikipedia 6:57 Computers have a hierarchy of storage 12:14 MediaWiki 12:15 Computers have a hierarchy of storage 14:54 Storage has limited bandwidth 19:39 Redundant Array of Independent Disks (RAID) 23:31 Basic idea of RAID 25:25 A Database Server @ NU 28:05 Large database servers 31:48 Persistent (disk) storage has alway...
CS-310 Lecture 07 - Load Balancing
มุมมอง 1.5K3 ปีที่แล้ว
Outline: 0:04 Last Lecture: Microservices, etc. 2:59 Load balancers 4:08 Basic idea of load balancer 5:04 Additional benefits of a load balancer 12:34 Two types of local load balancers 15:29 NAT Load Balancer 15:48 Nginx is can be used as a Reverse Proxy Load Balancer 16:58 Additional benefits of a Reverse Proxy LB 19:09 Comparison of Local load balancing options 20:43 Local load balancer limit...
CS-310 Lecture 06 - Microservices
มุมมอง 1.4K3 ปีที่แล้ว
Outline: 0:06 Last time: REST APIs and Data Serialization 2:41 Wikipedia is mostly one big PHP app 3:17 Cache and Database are “off-the-shelf” software 4:33 Monolithic apps 6:08 Advantages of a Monolithic design 12:58 Breaking up a monolithic architecture, example 16:06 Microservices 17:51 Microservice interactions 18:46 A few microservice disadvantages 24:08 Developing a service with a team 27...
CS-310 Lecture 05 - REST APIs and Data Serialization
มุมมอง 1.8K3 ปีที่แล้ว
Outline: 0:06 Last time: Proxies, and Caches 1:56 Application Programming Interfaces (APIs) 3:10 HTTP methods and responses 3:11 Application Programming Interfaces (APIs) 9:18 HTTP methods and responses 12:01 A weather information service (REST API) 15:37 Idempotence 21:05 REST API semantics must work with HTTP's rules 24:48 REST API example 30:22 Inputs and outputs of REST APIs 36:28 RESTful A...
CS-310 Lecture 02 - HTTP and Web Servers
มุมมอง 2.7K3 ปีที่แล้ว
CS-310 Lecture 02 - HTTP and Web Servers
Networking Lecture 18 - QUIC and Course Review
มุมมอง 1.1K3 ปีที่แล้ว
Networking Lecture 18 - QUIC and Course Review
Networking Lecture 15 - Encryption and Anonymity
มุมมอง 1.4K3 ปีที่แล้ว
Networking Lecture 15 - Encryption and Anonymity
Networking Lecture 13 - Medium Access Control
มุมมอง 1.4K3 ปีที่แล้ว
Networking Lecture 13 - Medium Access Control
Networking Lecture 10 - Router Internals and Routing Algorithms
มุมมอง 2.1K3 ปีที่แล้ว
Networking Lecture 10 - Router Internals and Routing Algorithms
Networking Lecture 08 - IPv4 Addressing
มุมมอง 2.1K3 ปีที่แล้ว
Networking Lecture 08 - IPv4 Addressing
❤
Thanks!
Thanks!
I love your lectures!
Thank you!
Hi, Steve! I wonder if possible to get the programming projects for this course? Thanks for your effort.
Sure, I posted the programming projects to my website: stevetarzia.com/teaching/ However you will need a Linux environment with Qemu to complete them, and this may be difficult to set up on your end.
Please offer the same for android users
Can i get this app or something close for androids ?
Hello! Unfortunately, this app is not available for Android. If you search for "badminton footwork" in the Google Play store you may find some similar apps, but the original app was just written for iOS. Happy training!
Thank you for posting these videos. I saw your post on HN and checked them out since I'm a professional dev but don't have much experience in distributed systems, and I just wanted to say you're an excellent teacher. It took me a bit to get through them, but you did a great job of providing the background rationale behind what motivated the development of a lot of these technologies in a way that I haven't been able to find in other places. Really appreciate it.
Thanks for your kind words, and best of luck applying these ideas to your work!
Great content Steve, thank you ! You mentioned that smartphone apps send notifications to the phone through 3rd party push notification service. Do they do that only when the app is offline, or is the method used when the user is online on the app as well ? For example, if a user is logged into Uber, just searched for a ride and is staring at the app ( like I do, lol) to see if he gets matched to a driver does Uber send me the ride match notification through 3rd party PNS as well ? Or is there a different mechanism ?
If the app is foregrounded, then the service can contact it directly, but often it will still use the 3rd party PNS so that the message will be delivered regardless of the user app state. In other words, it's easier to design the backend service to be unaware of whether the user is active in the app.
Thank you for these lectures!
Respected sir, at 40:35 , department_name in the student major table will refer to the primary key of the department table. You missed to represent this in the video. Thank you.
This is a good observation. We can follow your suggestion if we assume that "majors" are exactly equivalent to "departments." We cannot do this if there is not a one-to-one correspondence between departments and majors. For example, at Northwestern University the "Computer Engineering" major is associated with both the CS Department and the ECE Department.
@@SteveTarzia Yeah exactly. Can you please upload the slides of the classes or provide any link ? I am in deep need for your slides. Excellent content !! Thank you.
@@codecspy3479 Sure, the lecture slides are posted here: stevetarzia.com/teaching/
@@SteveTarzia Thanks alot sir.
sound is very low..
How does this work???
It's explained in this research article: stevetarzia.com/papers/ABS_mobisys.pdf
Thank you for scalability lectures.
This must be what comes after Scalable Systems class.
HAHAHAHA 4 years later this made me laugh
I love your videos
thanks
Terrific app Steve! Thanks a lot.
Steve, will this work with Voice Over to speak the location for the blind?
GREAT APP BUDDY! :) JUST DOWNLOADED MINE. BTW IS THAT FOOTWORK DONE IN A SQUASH COURT? AND IS IT GOOD ENOUGH? I'M ASKING BECAUSE I'M DOING THE SAME THING.
+James Cook Yes, that is a squash court! It's important that you learn the movements on a badminton court so that you know exactly how many steps to take and learn the proper size of the steps. However, once you learn the correct technique you can practice anywhere to improve your speed, reflexes, and endurance.
Actually, it was a *racquetball* court, which is slightly bigger than a squash court. However, a squash court is still bigger than half of a badminton court, so you should be alright using one.
alright man. thank's for the input! been training for almost 2 years now :)
Thanks! I did a few experiments to test whether speech "interference" would affect the system. We concluded that the system works as normal as long as there are short silent "gaps" between words. In other words, the room's acoustic fingerprint is not affected if a single conversation is occurring, but if you have lots of people talking over each other (imagine a cocktail party) then the acoustic fingerprint is affected.
This has so much potential, as far as adaptive technology for the visually impaired. Do you have any idea on how it reacts and how accurate it is with say people voices for example, on top of white noise?