Learn the fundamentals of database systems to understand and build performant backend apps Check out my udemy Introduction to Database Engineering database.husseinnasser.com
the only "efficient" (that I know of) way to keep count of rows, is to store the count somewhere in the database, that way u increase the number when you post something, decrease when u delete, etc. Startups have gotten huge bills for making large amounts of queries, just to get a count or get a sum of something.
As always in programming depends on the context, but yea I personally keep a count of of likes, favorites/post amount something like that in a particular structure and you increment or decrement the count. This is great for accuracy/precision and read time.
this is also slow in distributed databases I guess, because u have to update the count on every write, so only single write can happen on this type of db, because you have to take write lock on the count.. am I right?
Instagram used to have a cache to keep track of the follower count and then if the count was outdated, they would create a background job to do the select count(*) and then update the cache.
@8:41 Just do grades.forEach( g => gradeCalls.push( fetch(url).then(res => lblDuration.textContent = blahblahblah...) ); Promise.all(gradeCalls); Fetch just returns a promise, so you can chain an additional promise that updates the UI and push that to your gradeCalls. Now Promise.all() will execute the fetch calls AND the updates to the UI, which will happen after each fetch is completed instead of after all the fetch calls are completed.
@@OggerFN damn that's true.... if the UI updates need to be in sequential order, the time it takes to finish each fetch will affect the order of the updates.
For the question at 8:20, you can do a .then when you create the original promise and use Promise.all on the promises returned from then. So basically something like: gradeCalls.push( generateGradeCall().then( updateOnGradeChange ) ); Promise.all(gradeCalls).then( updateOnFinishEverything ); You can chain as many .then's as you want, since they return promises. It may not make for clean code though.
@@sadhlife Promise.all is settled when all the requests are successful. So my question is chaining a 'then' in fetch would call the callback on very successful request or when promise.all is settled?
The estimates are around 1.01 and 1.10 times as much as the actual values. You could actually then render some % (as a constant) of the estimates to align closer with the real values
Good catch! Can’t believe that slipped my mind. You are right! Since we are using HTTP/1.1 the browser will only open 6 TCP connections to the server and send those concurrently. The remaining 4 has to wait for some of the responses to come back in order to send the pending requests So my statement was not entirely accurate. Thanks!
The problem you mentioned in at 8:20, is actually really simple with JS promises! Basically right after the fetch() call, you would add a .then() where you can update each element. Since you still get a promise after calling .then(), the rest of your code with Promse.all will continue to work as-is. I just built a small demo to make sure it works, and here's the JS code I used: function apiPromise(elem) { return fetch('/api/', { method: 'GET' }) .then(response => { return response.json() }) .then(json => { elem.innerText = json.value }) } document.getElementById("button").addEventListener("click", function(event) { event.target.innerText = "loading" var outputElems = Array.from(document.getElementsByClassName('output')) outputElems.forEach(elem => { elem.innerText = "..." }) var promises = outputElems.map(apiPromise) Promise.all(promises).then(output => { event.target.innerText = "done" }) });
IMPORTANT COMMENT! Please read until the end. I have no experience with backend and I'm SQL newcomer but Veteran competitive programmer so there is a solution crossed my mind while watching the video. We want to count the number of records of a specific table efficiently and accurately, right ? Why not to make a small additional table this table will store the number of records of all our tables in the database. I said small because if we have 100 or even 10000 tables (actually I have no idea if this number is realistic or not) still small to make fast query on it. Our additional table (let call it Foo) has two columns the first column is maybe something like Table_id and the second column is Count (that stores the number of records of a table) So how that works ? Imagine we add new record in Grades table, then I will go to Foo table and go to the cell responsible for the Grades table and increase its value by the number of the new records we added in Grades table. And the same thing is done with deleting records. This idea is accurate and very fast because when we want to know the number of records of specific table we just make a query on Foo table which is small. Well, maybe it's difficult to update Foo table when delete many records with single query statement because (according to my knowledge) we don't know the exact number of deleted records But I think it works fine when counting the number of likes in social media apps Why? Cus no deleting many records with single query made on it. Just when some user click like add him to table and increase the number of records in Foo table by one. When click unlike delete it and decrease Foo by one. That works efficiently and give as the exact number of likes, right ? Finally, as I said I'm beginner with SQL so Please if there is something wrong with this idea correct me :) And sorry for my bad English
Well... it either way has to count all the current records in the given table to get precise number (and then it can follow your idea), but then comes the issue that there's no "read lock" - meaning, meanwhile you can add other records, so at the time of counting it will be different count. If we cannot lock the table like that, it will become an estimate either way (but more precise one).
The problem is that this only attempts to address querying the count of all records, when almost always what you're doing is counting some subset of records. For example, if we have a table post_likes that tracks when someone likes a post, knowing the total number of posts doesn't help me with knowing how many likes any particular post has gotten. The ANALYZE logic in pgsql is basically doing a more sophisticated version of what you're describing, with the added ability to give more than just total table size. EDIT: phrasing
What you just explained here is Caching but done in the SQL Database, you can also cache data in memory using KEY/VALUE pairs however Hussein's approach doesn't introduce any other layer of complexity because caching still requires another additional layer (Redis, Memcache ...etc).
I don't think this is a good idea. For the social media example, you will need one row for each post so if you have millions of posts you will a have a table with millions of rows which are constantly updated. Besides, you will not be able to use conditional statements in case you need to retrieve a specific part of the people who voted a post.
In my experience on a real scenario (100M+ rows for Pushpad), the estimates were completely wrong by several millions (much worse than your results). A real alternative / much better alternative is to keep the Visibility Map (VM) clean with frequent vacuum and then use index-only counts, which are fast. The problem is that it is not easy to always keep the VM perfectly clean. When a block is not clean PG goes to the heap, and this is the origin of all performance problems. There should be a way to tell Postgresql to just ignore the VM and consider all the index entries visible. Something like SELECT ESTIMATE(*) that considers all the index entries visible - without going to the VM or to the heap to check if they were modified or deleted recently.
@8:41 I could be wrong here, but I don't think the behavior you want is possible using a Promise (since they only resolve once... right?). Instead, maybe try using an Observable?
So the point is the query "explain (format jaon) select * from grades where g between $1 and $2". When we just use explain query, dbms just shows the query plan result with some estimated counts.
1. is there a reason you are not creating an index on the column you're filtering? without it you're essentially benchmarking sequential scan for (g between a and b), not count(*) per se. 2. the estimate approach without index kinda works only if you're lucky and the value-ranges you're counting are somewhat uniformly distributed. Consider this example, where I use a second column with a different distribution of values: postgres=# create table grades (id serial, g int, gg int); CREATE TABLE postgres=# insert into grades (g,gg) select 100*random(), (10*random())^2 from generate_series(1,15000000); INSERT 0 15000000 The query plan is not guaranteed to be a simple sequential scan, and if it's more complicated, the root node's "Plan Rows" value is not the estimate of how many rows the query would return. See sample select count(*) vs explain select * output with the non-uniform distribution: pastebin.com/C7ZUDFjZ
Good point! I forgot to create the index, on the g column. That would give better statistics and slightly better count performance on raw count. Thanks for sharing! Ill update the code to add the index
8:22 -----> fetch function returns a promise which can be chained with a `then` to update the UI for that specific API call. Guess what the return type of the chained `then` is? It's a promise itself with a generic type of whatever is returned from the `then` function. The chained `then` then can be called over Promise.all([array of then promises]) to be called together.
I don't think there's a need for compromise here: There are good tools for handling fast changing values accurately and much faster. Why not use an in memory database like Redis for this?
For an application like instagram for example where the likes could get rly big, is keeping that number as a table property that gets incremented on each like a valid solution for this problem ?
you can do increment and decrement of the sum on another table every time grade has added / changes, so then you can just do select sumOfGrade from gradeSumTable
That works but slows down your writes and throughput as a result. Not to mention that you have to introduce locking to the sum to avoid race conditions issues between concurrent transactions, which slows writes more.
I would fix your problem at 8:20 by doing something like: Promise.all( ['0-10', '10-20', '30-40' ].map(fetchAndUpdateGrades)) async function fetchAndUpdateGrades( grade ) { // fetching data for a given grade const result = await fetch('localhost:8000/'+grade); // updating UI as soon as I get data back from server for this grade document.getElementById(grade).innerText = result.data; }
Good fix! someone created a pull request based on your fix and I just merged it. The trick really is to simply execute the fetch, that returns a promise, execute JSON() that returns a promise to parse the HTTP body into JSON. then update the label and then return a resolved promise. All those resolved promises then pool into Promise.all and that did the trick! github.com/hnasr/javascript_playground/pull/21/commits/2e4b6bbab57781f6d21f118984097aa9a267d45d
hey hussein! your channel is a gold mine! I wish i stumbled upon your videos earlier in my career. I'm having a job interview for a database admin position later this week. do you have any advice or tips that can give me?
I suppose that view counts is an actual programmed counter that increase by one each time a user view the source. For things like Facebook the like count shows instantly but it has the feature of "people who likes this" that will take longer to load a small part of the "people who likes this" accounts.
Hmmm. What about indexes? Assuming you have a primary key on "id" and an index on "g", I believe you could execute the following queries very fast without any problem: select count (id) from grades; select count (g) from grades where g between 7 and 9; This is supposed to be faster because the index already contains statistics about the row count
As I know we can't use this trick in SQL Server I mean with sql server we can do this like this : set showplan_all on; go; select count(*) from table; but this is TSQL and we cant't use it in as a query in web apps
Thanks 🙏 but trace or any returns the first response that comes back which won’t give accurate duration of how long all of the requests took. I guess I can provide the duration for each request and sum all the results on the client
Hi Hussein I just came accross your channel and fell in love with the content I request to you to make project including webrtc, websocket nodejs and react or angular which can outstand resume please do it's a humble request
Actually there are benchmarks showing that in some cases postgres can be really really slow compared to other databases for count(*). Search for count(*) performance postgres vs SQL server in Google. Postgres can be as much as 400 times slower.
Yes can use trigger for that. But kinda difficult to get the where condition. so what i did is create trigger for follow/unfollow and increase/decrease the number by 1 on the follower column at the user table.
I was thinking you could have variables in your database for the counts and increment them whenever a new object gets added to a collection, but I couldn't figure out an easy way to do this since I don't think you can really store global variables in the database. I think it would be good if you needed the count often, it would sacrifice a little bit of time when an object gets added, but not too much at all. edit: I guess you could have a counts collection and each object would have the name of the count and tbe count itself
You could do something like what you're suggesting with triggers. You could also perform a select count and store the result as part of a materialized view or some similar form of cached data.
Hello Hussein. Hope you're great, I was just wondering if you could make a video on ESB and its implementation. That could be great to have a good understanding of it. Thanks in advance.
He can estimate it faster, he didnt say he can get the accurate data faster. And if the hypotethical example of data between ages, if its actually mattered ofc the data model would be different, the purpose of the video is never about the actual data anyways
something like (likes, views, comments count) should be on a nosql database it's easy when someone likes u add it to database and fire event to add to Rabbitmq for example to update nosql database to increment the count by 1 for example
Why particularly NoSQL? If I'm understanding you, it seems like you're just proposing maintaining real time counts elsewhere, which can be done a number of ways
Promise.all will wait for all promises to be fulfilled, But Individual Promises don't care about other Promises and should run the callback as soon as they are fulfilled and process is in the possition. there may be other way of doing this in one call but i don't know them.
Why don't you do both queries and show the estimate first to the user with some feedback showing that it is still in progress and only when the precise query finishes you put a nice number transition between the estimate and the real number?
For more accurate estimates would it be efficient to run an estimate and then run queries around that number to see what returns a value. Like, if we have 100 rows and it estimates 110, test 120 and 90, 90 returns something so go in the middle and try 100, 100 returns something so go in the middle and try 105, 105 didn't return so you can keep going to be more precise or estimate the the true value to be between 100-105. I'm just getting into back end and I haven't used any kind of database, so I don't know if that would be any good, but I feel like the db might be able to query an exact row very quickly, right?
just my thinking about way to achieve the logic you want in minute 9. been a while since last time i used js for code something but this logic should be sufficient to achieve that goal use loop like for, or maybe for of, or for in for (let i = 0; i < grades.length; grades++) { let element = grades[i]; element.then(result => /* DO SOMETHING HERE*/); } /* DO SOMETHING HERE*/ will be executed for every success request... is what i think will happen
Promise.all is used when your program needs the result of all the promises to continue, saving you from resolving the next promise within the previous one (eew) In this case you should remove promise.all and use forEach on gradeCalls to call then on each promise individually to have them resolve as they do.
Uhm ? why not just cache the freaking count in some other statistics table or in memory or smth and update it based on some actions or intervals? EDIT: Yea true ESTIMATING does not introduce any complexity but it doesn't show accurate numbers, to be honest, I think the best solution to this problem is a combination of the two solutions. You can *"ESTIMATE"* when there is no cache and then just proceed to generate the cache after sending the estimate to the user in memory (to reduce complexity).
The correct way to fix the performance is to add indexes, based on the queries performed. Such as your grades. Using Estimate is not the correct way to get accurate results.
Learn the fundamentals of database systems to understand and build performant backend apps
Check out my udemy Introduction to Database Engineering
database.husseinnasser.com
yo he got them emotes 🥶🥶🥶🥶
That is the most beautiful frontend app I've seen in my entire life.
I'm merely mesmerized!
That UI is on the latest framework called HTML. I've heard rumors it's more efficient then react and vue.
You should get out more.
So you're a backend right?
@@ipostfacebookvideo2989 sadly no, I'm just a person
the only "efficient" (that I know of) way to keep count of rows, is to store the count somewhere in the database, that way u increase the number when you post something, decrease when u delete, etc.
Startups have gotten huge bills for making large amounts of queries, just to get a count or get a sum of something.
As always in programming depends on the context, but yea I personally keep a count of of likes, favorites/post amount something like that in a particular structure and you increment or decrement the count. This is great for accuracy/precision and read time.
this is also slow in distributed databases I guess, because u have to update the count on every write, so only single write can happen on this type of db, because you have to take write lock on the count.. am I right?
Instagram used to have a cache to keep track of the follower count and then if the count was outdated, they would create a background job to do the select count(*) and then update the cache.
@8:41 Just do
grades.forEach( g => gradeCalls.push( fetch(url).then(res => lblDuration.textContent = blahblahblah...) );
Promise.all(gradeCalls);
Fetch just returns a promise, so you can chain an additional promise that updates the UI and push that to your gradeCalls. Now Promise.all() will execute the fetch calls AND the updates to the UI, which will happen after each fetch is completed instead of after all the fetch calls are completed.
doesn't matter too much as the calls are taking similarly long
@@OggerFN damn that's true.... if the UI updates need to be in sequential order, the time it takes to finish each fetch will affect the order of the updates.
For the question at 8:20, you can do a .then when you create the original promise and use Promise.all on the promises returned from then. So basically something like:
gradeCalls.push( generateGradeCall().then( updateOnGradeChange ) );
Promise.all(gradeCalls).then( updateOnFinishEverything );
You can chain as many .then's as you want, since they return promises. It may not make for clean code though.
async await ftw
The updateongradechange would be called when all the promises are resolved
Promise.race or any would be the go
@@mthaha2735 nope, updateOnGradeChange will be called on each request received. upgradeOnFinishEverything is what will be called at the end
@@sadhlife Promise.all is settled when all the requests are successful. So my question is chaining a 'then' in fetch would call the callback on very successful request or when promise.all is settled?
The estimates are around 1.01 and 1.10 times as much as the actual values. You could actually then render some % (as a constant) of the estimates to align closer with the real values
was thinking that
@@guiAI seems a bit hacky but if you did some more digging into it, I'd say you'd be able to find a reliable constant to use
@@CallumAtwal i think just getting it a bit down would be enough, the number wouldnt be right anyways, but it would be "righter"
This is nice, although I can't think of any situation where an estimate would be good enough when looking for the actual count.
@Hussien - i have a doubt as you are sending 10 requests in parallel, aren't browser limit of 6 connect per host queue the other 4 requests?
Good catch! Can’t believe that slipped my mind. You are right! Since we are using HTTP/1.1 the browser will only open 6 TCP connections to the server and send those concurrently. The remaining 4 has to wait for some of the responses to come back in order to send the pending requests
So my statement was not entirely accurate. Thanks!
I was so close to running select count(*) today but remembered this video and had to come back and steal this! Thanks a lot
The problem you mentioned in at 8:20, is actually really simple with JS promises! Basically right after the fetch() call, you would add a .then() where you can update each element. Since you still get a promise after calling .then(), the rest of your code with Promse.all will continue to work as-is.
I just built a small demo to make sure it works, and here's the JS code I used:
function apiPromise(elem) {
return fetch('/api/', {
method: 'GET'
})
.then(response => {
return response.json()
})
.then(json => {
elem.innerText = json.value
})
}
document.getElementById("button").addEventListener("click", function(event) {
event.target.innerText = "loading"
var outputElems = Array.from(document.getElementsByClassName('output'))
outputElems.forEach(elem => {
elem.innerText = "..."
})
var promises = outputElems.map(apiPromise)
Promise.all(promises).then(output => {
event.target.innerText = "done"
})
});
I want to learn more about this, got any helpful links?
IMPORTANT COMMENT!
Please read until the end.
I have no experience with backend and
I'm SQL newcomer but Veteran competitive programmer so there is a solution crossed my mind while watching the video.
We want to count the number of records of a specific table efficiently and accurately, right ?
Why not to make a small additional table this table will store the number of records of all our tables in the database. I said small because if we have 100 or even 10000 tables (actually I have no idea if this number is realistic or not) still small to make fast query on it.
Our additional table (let call it Foo) has two columns the first column is maybe something like Table_id and the second column is Count (that stores the number of records of a table)
So how that works ?
Imagine we add new record in Grades table, then I will go to Foo table and go to the cell responsible for the Grades table and increase its value by the number of the new records we added in Grades table. And the same thing is done with deleting records.
This idea is accurate and very fast because when we want to know the number of records of specific table we just make a query on Foo table which is small.
Well, maybe it's difficult to update Foo table when delete many records with single query statement because (according to my knowledge) we don't know the exact number of deleted records
But I think it works fine when counting the number of likes in social media apps
Why?
Cus no deleting many records with single query made on it.
Just when some user click like add him to table and increase the number of records in Foo table by one. When click unlike delete it and decrease Foo by one.
That works efficiently and give as the exact number of likes, right ?
Finally, as I said I'm beginner with SQL so Please if there is something wrong with this idea correct me :)
And sorry for my bad English
Well... it either way has to count all the current records in the given table to get precise number (and then it can follow your idea), but then comes the issue that there's no "read lock" - meaning, meanwhile you can add other records, so at the time of counting it will be different count. If we cannot lock the table like that, it will become an estimate either way (but more precise one).
The problem is that this only attempts to address querying the count of all records, when almost always what you're doing is counting some subset of records. For example, if we have a table post_likes that tracks when someone likes a post, knowing the total number of posts doesn't help me with knowing how many likes any particular post has gotten. The ANALYZE logic in pgsql is basically doing a more sophisticated version of what you're describing, with the added ability to give more than just total table size.
EDIT: phrasing
What you just explained here is Caching but done in the SQL Database, you can also cache data in memory using KEY/VALUE pairs however Hussein's approach doesn't introduce any other layer of complexity because caching still requires another additional layer (Redis, Memcache ...etc).
I don't think this is a good idea. For the social media example, you will need one row for each post so if you have millions of posts you will a have a table with millions of rows which are constantly updated. Besides, you will not be able to use conditional statements in case you need to retrieve a specific part of the people who voted a post.
he literally never has a dislike on this content !! the best teacher for a reason.. :) Thanks for sharing
Nah, it's just estimate number from TH-cam, the actual number always 0
@@your_skyfall u nailed it kind of. but that is my point too no dislikes 🙈
Had to reopen the video today. To actually use the explain query.
Thanks @Nasser
This channel is a gold mine . I get gold every time I visit
Send me some
In my experience on a real scenario (100M+ rows for Pushpad), the estimates were completely wrong by several millions (much worse than your results). A real alternative / much better alternative is to keep the Visibility Map (VM) clean with frequent vacuum and then use index-only counts, which are fast. The problem is that it is not easy to always keep the VM perfectly clean. When a block is not clean PG goes to the heap, and this is the origin of all performance problems. There should be a way to tell Postgresql to just ignore the VM and consider all the index entries visible. Something like SELECT ESTIMATE(*) that considers all the index entries visible - without going to the VM or to the heap to check if they were modified or deleted recently.
@8:41 I could be wrong here, but I don't think the behavior you want is possible using a Promise (since they only resolve once... right?). Instead, maybe try using an Observable?
So the point is the query "explain (format jaon) select * from grades where g between $1 and $2". When we just use explain query, dbms just shows the query plan result with some estimated counts.
as soon as you click on the estimate button NATIONAL GEOGRAPHY channel pop up on my mobile screen 😂😂😂
1. is there a reason you are not creating an index on the column you're filtering? without it you're essentially benchmarking sequential scan for (g between a and b), not count(*) per se.
2. the estimate approach without index kinda works only if you're lucky and the value-ranges you're counting are somewhat uniformly distributed.
Consider this example, where I use a second column with a different distribution of values:
postgres=# create table grades (id serial, g int, gg int);
CREATE TABLE
postgres=# insert into grades (g,gg) select 100*random(), (10*random())^2 from generate_series(1,15000000);
INSERT 0 15000000
The query plan is not guaranteed to be a simple sequential scan, and if it's more complicated, the root node's "Plan Rows" value is not the estimate of how many rows the query would return.
See sample select count(*) vs explain select * output with the non-uniform distribution: pastebin.com/C7ZUDFjZ
Good point! I forgot to create the index, on the g column. That would give better statistics and slightly better count performance on raw count.
Thanks for sharing! Ill update the code to add the index
I really want to know a solution for the question at 8:22
8:22 -----> fetch function returns a promise which can be chained with a `then` to update the UI for that specific API call. Guess what the return type of the chained `then` is? It's a promise itself with a generic type of whatever is returned from the `then` function. The chained `then` then can be called over Promise.all([array of then promises]) to be called together.
your frontend is a masterpice
I don't think there's a need for compromise here: There are good tools for handling fast changing values accurately and much faster. Why not use an in memory database like Redis for this?
For an application like instagram for example where the likes could get rly big, is keeping that number as a table property that gets incremented on each like a valid solution for this problem ?
Super video! I applauded for R$10.00 👏👏👏
❤️❤️🙏
you can do increment and decrement of the sum on another table every time grade has added / changes, so then you can just do select sumOfGrade from gradeSumTable
That works but slows down your writes and throughput as a result. Not to mention that you have to introduce locking to the sum to avoid race conditions issues between concurrent transactions, which slows writes more.
I would fix your problem at 8:20 by doing something like:
Promise.all( ['0-10', '10-20', '30-40' ].map(fetchAndUpdateGrades))
async function fetchAndUpdateGrades( grade ) {
// fetching data for a given grade
const result = await fetch('localhost:8000/'+grade);
// updating UI as soon as I get data back from server for this grade
document.getElementById(grade).innerText = result.data;
}
yup! let's teach mr. hussein some front-end tips while we learn about backend :)
Good fix! someone created a pull request based on your fix and I just merged it.
The trick really is to simply execute the fetch, that returns a promise, execute JSON() that returns a promise to parse the HTTP body into JSON. then update the label and then return a resolved promise. All those resolved promises then pool into Promise.all and that did the trick!
github.com/hnasr/javascript_playground/pull/21/commits/2e4b6bbab57781f6d21f118984097aa9a267d45d
@@hnasr Who would have thought one day TH-cam comments sections could replace stackoverflow. People would copy paste code from youtube comments.
Pure Gold! Shukran :)
hey hussein! your channel is a gold mine! I wish i stumbled upon your videos earlier in my career. I'm having a job interview for a database admin position later this week. do you have any advice or tips that can give me?
can we do same thing in MySQL?
All databases store stats.
@@diehardfighter 👍
I suppose that view counts is an actual programmed counter that increase by one each time a user view the source. For things like Facebook the like count shows instantly but it has the feature of "people who likes this" that will take longer to load a small part of the "people who likes this" accounts.
Hmmm. What about indexes? Assuming you have a primary key on "id" and an index on "g", I believe you could execute the following queries very fast without any problem:
select count (id) from grades;
select count (g) from grades where g between 7 and 9;
This is supposed to be faster because the index already contains statistics about the row count
But why is count so slow in Postgres while it can be extremely fast in other dbs ?
8:41 I think what you want is .then() syntax
As I know we can't use this trick in SQL Server
I mean with sql server we can do this like this :
set showplan_all on;
go;
select count(*) from table;
but this is TSQL and we cant't use it in as a query in web apps
Thanks. Very interesting video 🔥
Good video! If you want to return a promise when any promise in an array resolves use Promise.any or Promise.race instead of Promise.all 👍
Thanks 🙏 but trace or any returns the first response that comes back which won’t give accurate duration of how long all of the requests took. I guess I can provide the duration for each request and sum all the results on the client
@@hnasr yeah just a reduce on the array of promises returned by promise.all should do the trick then!
Nice , I love to review code , going through your functions makes me feel cool 🌹🌹
Fill cool?
@@tqwewe I mean feel 😅
Hey man can you do some videos on big data and realtime analytics (streaming). Thanks
How would it be useful to search for bad data?
How exactly is this estimate calculated?
Hi Hussein I just came accross your channel and fell in love with the content I request to you to make project including webrtc, websocket nodejs and react or angular which can outstand resume please do it's a humble request
Does other DBMS suffers from the same performance issue?
Yes all DBMS has the same thing, its just slow to count millions of rows, indexes help but the more rows you have to slower it gets to count.
Actually there are benchmarks showing that in some cases postgres can be really really slow compared to other databases for count(*). Search for count(*) performance postgres vs SQL server in Google. Postgres can be as much as 400 times slower.
How about manually storing the count and incrementing on insert?
Yes can use trigger for that.
But kinda difficult to get the where condition.
so what i did is create trigger for follow/unfollow and increase/decrease the number by 1 on the follower column at the user table.
You can use promise.any
syntax shaming, this guy cracks me up
does Postrges update the statistics after every DML automatically?
Why not use hyper log log? It would seem to be more accurate.
best way to create real-time db using mysql ?
What If I need actual results in less time ?
maybe keep the likes count somewhere else on every like event?
I was thinking you could have variables in your database for the counts and increment them whenever a new object gets added to a collection, but I couldn't figure out an easy way to do this since I don't think you can really store global variables in the database. I think it would be good if you needed the count often, it would sacrifice a little bit of time when an object gets added, but not too much at all.
edit: I guess you could have a counts collection and each object would have the name of the count and tbe count itself
Yes that works but slows down your writes as a result
You could do something like what you're suggesting with triggers. You could also perform a select count and store the result as part of a materialized view or some similar form of cached data.
Hello Hussein. Hope you're great, I was just wondering if you could make a video on ESB and its implementation. That could be great to have a good understanding of it. Thanks in advance.
Thanks! I learn something today.
You can use for await loop
how can i join and get access to ur members-only video content? I couldn't figure it out lol
THANK YOU BROS ! . I NEED IT
for features involving getting metrics and displaying, ElasticSearch is the way to go... Using SQL DB's for these purposes are kind'a outdated.
7:38 esh-ti-em-el
2:45 That's , as they say, low-key hilarious. :)
promise question: you should be able to do this:
grades.forEach( (g,index )=> fetch('...').then((a)=>{ /* set cell with index here */}))
You are looking for Promise.race() or Promise.any()
Thats why you should index your database properly
Make the output in a graph with each a percentage of all I doubt anyone would notice the difference :-)
So you were saying, you can reach wrong data very fast ? Thats amazing man. Good work, keep it up!
He can estimate it faster, he didnt say he can get the accurate data faster. And if the hypotethical example of data between ages, if its actually mattered ofc the data model would be different, the purpose of the video is never about the actual data anyways
Summary and Detail 😉
The space between function name and ( is bothering me
something like (likes, views, comments count) should be on a nosql database it's easy when someone likes u add it to database and fire event to add to Rabbitmq for example to update nosql database to increment the count by 1 for example
That's I was thinking
Why particularly NoSQL? If I'm understanding you, it seems like you're just proposing maintaining real time counts elsewhere, which can be done a number of ways
@@Mike10131994 sure, not particularly nosql, but I think is more easy set in your "Post" model a field with a likes counter.
@@Mike10131994 i mean like redis
Select * and select count(*) are different though
Promise.all will wait for all promises to be fulfilled, But Individual Promises don't care about other Promises and should run the callback as soon as they are fulfilled and process is in the possition. there may be other way of doing this in one call but i don't know them.
Why don't you do both queries and show the estimate first to the user with some feedback showing that it is still in progress and only when the precise query finishes you put a nice number transition between the estimate and the real number?
Syntax Shamming 😂😂😂
The problem here is that the estimate count changes every time you query the database
Sad you didn't use Svelte
Aggregation with mongodb works very well in such cases.
But in aggregating too we have $count stage which I think is same as Select count(*). I don't think aggregation solves this problem.
You could use web socket for the response, updating individually each cell
I think you are looking for Promise.any()
ما شاء الله لا قوة إلا بالله
بارك الله فيك
*ahem* the entire HTML community stopped using table elements. A lot of specs that had to do with table were deprecated
For more accurate estimates would it be efficient to run an estimate and then run queries around that number to see what returns a value. Like, if we have 100 rows and it estimates 110, test 120 and 90, 90 returns something so go in the middle and try 100, 100 returns something so go in the middle and try 105, 105 didn't return so you can keep going to be more precise or estimate the the true value to be between 100-105. I'm just getting into back end and I haven't used any kind of database, so I don't know if that would be any good, but I feel like the db might be able to query an exact row very quickly, right?
just my thinking about way to achieve the logic you want in minute 9.
been a while since last time i used js for code something but this logic should be sufficient to achieve that goal
use loop like for, or maybe for of, or for in
for (let i = 0; i < grades.length; grades++) {
let element = grades[i];
element.then(result => /* DO SOMETHING HERE*/);
}
/* DO SOMETHING HERE*/ will be executed for every success request... is what i think will happen
react angular bootstrap tailwind tutorial when?!?
i am just gonna warehouse the count.
Promise.all is used when your program needs the result of all the promises to continue, saving you from resolving the next promise within the previous one (eew)
In this case you should remove promise.all and use forEach on gradeCalls to call then on each promise individually to have them resolve as they do.
LOL 7:25 yep it does... it does LOOOOVE json ♥
Nice!
great idea
Nice.
I'm a front-end guy, myself. 😉
Uhm ? why not just cache the freaking count in some other statistics table or in memory or smth and update it based on some actions or intervals?
EDIT: Yea true ESTIMATING does not introduce any complexity but it doesn't show accurate numbers, to be honest, I think the best solution to this problem is a combination of the two solutions. You can *"ESTIMATE"* when there is no cache and then just proceed to generate the cache after sending the estimate to the user in memory (to reduce complexity).
Caching works but it introduces complexity. This is a bare simple non-cache solution
The hype about so many frontend languages for the table design was a joke folks!
Yeah uhm it took me 3 min to subscribe to you :>
The correct way to fix the performance is to add indexes, based on the queries performed. Such as your grades. Using Estimate is not the correct way to get accurate results.
While indexes help make query fasters, it will still be relatively slow to count million of rows.
XD ur html syntax
Woow I didn’t know that.
very well.
Hahahaha best Web design award
you can do SELECT COUNT(specific_field)
than SELECT COUNT(*) because it select all fields in the table...
Does not change anything in Postgres. Actually count(*) is generally faster. The * in count(*) does not mean 'all columns'
Not worth the effort. Count is fine.
You can get actual count in 300ms.
how?
how?
how?
how?
ahh 😏... maintain the count in dedicated column, then fetch that