Maps, Arrays and Subcollections, Oh My! | Get to know Cloud Firestore #4
ฝัง
- เผยแพร่เมื่อ 7 ต.ค. 2024
- Is your Cloud Firestore data structure set up correctly? Cloud Firestore supports Subcollections, Maps, and Arrays, all of which can be useful, but it's sometimes difficult to know when to use which one. Why create a subcollection when you could just create an array of maps? Or a map of maps? When is it appropriate to use one over the other? And what would work best in your app?
In this episode of Get to Know Cloud Firestore, Todd does his best to get your questions answered and gives you all the pros and cons, so you can structure your data the right way.
Cloud Firestore Data model → bit.ly/2PZtQnD
More episodes on Cloud Firestore → goo.gle/Cloud-...
Subscribe to Firebase → goo.gle/Firebase
![โรงเรียนตอนตี 2 โคตรหลอน..!! [โรงเรียนไดเม่]](http://i.ytimg.com/vi/2MAEMwqVYh8/mqdefault.jpg)
You guys have mastered the art of teaching.
The animations are amazing
So true
NOOOO.. we learn by seeing the codes. Show us examples.. we want the codes..
This is one of the best explanations I've ever seen on data structures. Not just for firebase, but noSQL in general. Thank you.
love the pace of these videos. All teaching videos must be made like this.
I swear this guy is the only part of any google software I like.
So the basic rule of thumb might be "Shape your data like your queries". Data that are usually queried together should go together and hierarchies that exist in your app should probably also be mirrored in the data.
Hey todd from 2019 you forgot to make a new caption for the video after edit
Ayman Shaaban ELsayed DEVELOPER this was really impressive how they did this!
you can also switch to auto generate caption
Great video, thank you! I'm just learning and designing my NoSQL model and every one of these videos teach me exactly what I wanted to know. I have one complain: when you update the original video please update the subtitles too!
"This tends to put your app in a state I like to call 'uninstalled'" hahahaha
getting uninstalled
This is a very good explanation for Firestore.
Now I know which I should use or not use on Cloud Functions
by knowing these rules on Firestore. I think I can learn more by watching the whole series.
(2018, 2019, 2020...) Todd is cool in any year. Thank you for this video!
Security rules need to be taken into account when setting up data structures as well. I just found this out the hard way. Luckily I'm still in the dev phase so can pivot to a different data structure that allows me to set different security rules for sub-collections vs the parent collection.
Great explanation! I LOVE that you also update the videos! That's great!
Could you please next time cover topic of timestamps, dates in Firestore / Cloud functions! I mean getting, storing and sending to clients! Thanks in advance!
,so
This is the most important video about Firestore imo.
So Todd can time travel. great work google 👍
Love love love love love all these Firebase videos! Awesome! Keep going!
Thanks Todd, I REALLY appreciate these insights on Cloud Firestore.
This guy is just brilliant and the ending was so funny.
Amazing video! That ending was HILARIOUS
Very helpful video. Thanks!
We really appreciate the positive feedback, Philippe! Thanks for watching 😀
If you liked this, be sure to check out more episodes on Cloud Firestore here: goo.gle/Cloud-Firestore
I find that using sub-collections are great for data privacy because you can have different rules for a document and its sub-collections. Good data structures plus strong rules equals wicked privacy kung fu.
This firestore video series is so fantastic 👌
Thanks for array-contains. This is really good!
Just keep them videos coming!!!
1 - 1 mb per document
2 - you can not retrieve data partially
3 - Quering are shallow
4 - better put ur data top lvl if u want search thru all document or retrieve
Great teaching experience guys, great job Todd. The only thing that i am skeptical all this time with Firestore is the limitations of upgrading. Let me explain better. We build an app today with specific needs, let's say for example a Charles Dickens' app where we need to get the books titles only. After users feedback we decide to upgrade our app and add "search by character's occupation" function. In these changes (upgrades) of our app, we need to re-design our entire db "schema" and maybe migrate old db structure to a new one... It sounds tricky
Geopoint will have their own querys soon? Or not at all?
Like distance between geopoints or querys by proximity?
Really waiting for those!
I would be interested in this too as I have a project that might need to calculate the distance between points. I am dreading having to look up that crazy formula and trying to figure out how make it work. The Geopoint object has a compareTo method that might be used for this, but the docs don't really elaborate on what it returns: firebase.google.com/docs/reference/android/com/google/firebase/firestore/GeoPoint#compareTo(com.google.firebase.firestore.GeoPoint)
I’m running a side Postgres dB with one table that just keeps track of the Firestore Id and a point lat long. So I query the Postgres when I need geospatial but other 99% is direct to firestore
July of 2020, this is still not supported officially, but, i got filter my documents adding they on Adapter and using Collections.sort(); But, the disavantage is the realtime data not appears more.
Clarified a lot of things, tysm
Thanks, it really helped!
You're welcome!
Awesome! Congratz to the team
Great video, thanks
Glad you liked it!
Cool as always! Cheers from Slovakia!
You're a superb teacher!
Thanks for the quick education!
Excellent tutorial
Glad you liked it
5:50 Aren't 2 million reads = 1.20$?
yes and 2 billion reads are 1200$
fawk !
Right On Time.. Again! Keep it up!
It could be nice if we could check if a date or number is what we expected(in rage or higher or lower) into a object in an array
It's very optimize way to store data in collection and send/receive in JSON.
Thank you so much!
this guy is awesome
🤣 the shark tank refrence at the end
ikr, that cracked me up
Absolutely Great Video ! Firebase is Awesome
Is 8:54 a typo? I'm guessing that both subcollections should be named "characters". Thanks, this video helps!
I'm surprised the video didn't cover one big limitation of ArrayContain query: they cannot be chained.
I would love to have a pint with Todd!
I have the answer and your back in I promise! Dickens and all the rest.
I was asking myself this question just yesterday bcz we need to scale up our app.
I keep thinking just use firestore for realtime updates but use another database for real search capability and return ids for firebase to use. Use the right tool for the job but requires keeping data in sync.
The distinguishing feature of arrays is index-based access, which Firebase doesn't support. Seems like a better name for "arrays" would have been "sets" (or perhaps "multisets" or "bags" if repeated values are supported). Or am I missing something?
In Firestore rules the arrays are called lists and object are called maps. I think lists would be a more accurate description of them, I am not sure why the difference in terminology between the data structure and the rules... maybe because arrays and objects are more web dev terminology and lists and maps are more "traditional" terminology (at least from my perspective) and probably more correct from a CS point of view.
Interesting. Like arrays, the abstract data type "list" corresponds to a number of ordered values. Lists usually support, at a minimum, ordered traversal of elements (i.e., a "first" and "next" operation) as well as ordered insertion (e.g., "append", "prepend"). From what I gathered from the video and my (admittedly thin) knowledge of the API, Firebase doesn't support any order-based operations on what they call arrays (in this video) or lists (in the rules). Therefore, I don't think "list" is any more appropriate than "array" for what Firebase does offer. If they were to ask me, I might have suggested they use "map" and "collection", but they already assigned a different meaning to "collection", so that's out. Oh well.
Iteration is supported, its just not supported on the database (as far as I am aware). If you use the Javascript SDK an array from the database is an array object, and a Javascript array is a list. Also the support methods of a data structure do not make it a list or not, the actual structure determines that. Another also: when you query a document you get the ENTIRE document, so there is no real need to iterate over an array or manipulate it on the database, you would do that in whatever code made the query (be it client side with a client SDK or API or server side with the admin SDK or API). For example, if you wanted to do server side manipulation you would do that with Cloud Functions using the admin SDK, not in Firestore itself.
Thanks for the info, Todd. What you're describing sounds to me like a set implemented as an array which is in turn simulated using a map. But the exposed functionality sounds like a set, not an array or a map. Yes, values are entered by supplying them in some order (that looks like an array) and they are returned in some order (that looks like an array). However, from what I've read, duplicate values are not supported (the array ["a", "b", "c", "a", "b", "c"] gets stored as ["a", "b", "c"]). In this case (and in others) there's no order guarantee. From the api user's perspective, it doesn't have the characteristics one would expect of an array.
I'm not sure I understand your comment. On the one hand you say I have indexed-based access and ordering, but on the other hand I don't have that because the only operators I have don't allow for indexed-based access or ordering. (For instance, I can't insert an item at position 3 if the same value already exists at position 0. Actually, I can't insert an item at position 3 at all, since the api doesn't allow it.)
I understand from the blogs why the api is designed this way, and I think it's a good design. A true array model would have lots of issues. I just object to it being called an array when the only thing it has in common with an array is that sets of values are transmitted in both directions in an apparently ordered form (which is not guaranteed to be preserved).
NICE TEACHING
Nice video but it would be great to have an article to accompany it
As a hobbyist with no professional background, this was immensely helpful! I was wondering the use case of subcollections were if we could just map them into the top level document (I used them anyway for better organization). Especially since one of the key features of nosql is to denormalized data, this was highlighted in video 1 as well ithink.
I am currently using a subdocument (e.g,"users/blogs") to copy a top level document (e.g.,"blogs") into a "users" document but currently need 2 api calls to retrieve user and the user's blogs data, a userDocRef and a userBlogsRef. I will be rewriting my code to make it so that the user's blog data is in the user's document and reduce the number of server calls when retrieving data. Posting/updating data will unlikely be improved.
(btw, its not really a blog app im making, it's an operations (management) program but seemed like blogs would be easier to understand).
I wouldn't get to focused on reducing database calls, but if you wanted to make things quicker and reduce calls you could do something like this:
Have an array in the Users document that lists basic details of each blog, say the document ID, the title and extract. Then have the full blog documents in a sub collection. That way when you retrieve the User document you get a list of blog posts without the second query but still have enough detail to know exactly which blog document to query if you need to. Then maintain consistency using cloud functions to update the array when ever a blog is created or updated. Here are the docs on those triggers: firebase.google.com/docs/functions/firestore-events
Good video, now let's wait 3 moths for the next one :O
@@ToddKerpelmanCorp it was all a lie :C
Thank u for this awesome video
hey, guys, just typing this to say your "subscribe" and "watch more" linked windows are popping up way too early - that may be because of your new content addition ;)
anyway, a great video collection, thanks!
Awesome info...
Caught a small error at 11:24, missing a "s" after "keyword". Thanks for making this btw.
So, arrays should be used like sets?
I think they just index the arrays as objects. I just tried adding duplicate strings to the array and it worked just fine.
Great video. I recently ran into the current limitation of not being able to delete collections (or sub-collections) so deleting a document with sub-collections would orphan the sub-collections. Seems like another reason to use arrays within documents instead of sub-collections - at least for now. Your thoughts?
Which among these is more efficient? Updating an array field inside a document or listener on a sub-collection with documents?
Now we need Todd from 2022
Nice
When to use simple string to store a reference vs a real reference (type) ?
So, in the end, what's the best data structure for an Agenda like application to store customers appointments 😄😄😄? The app have a collection of customers and I need to show the full month appointments in a "Google calendar style", but also need to count how many times a client has an appointment with me. Any suggestion? Thanks a lot.
Hey, thanks for the guidelines/rules. But a question here on arrays in documents: How do I validate the data entered to the firestore. How can I define the type of the elements that get inserted via the security rules? For example, if I only want to have objects inside an array with some specific values.
Security rules does have some support for this, but its not extensive. Depending on your use case I would suggest security rules as a first line and cloud functions as a second line. If it is critical stuff then you can disallow writes in the rules and create a cloud HTTP function that parses a request and writes to the database directly (the admin SDK bypasses rules), more detail here: cloud.google.com/functions/docs/writing/http
I don't know what is the best way of creating a voting system for example for "movies". I want people to vote from 1 to 5 stars but also keep track if they voted or not.
I'm showing the vote count number and score of each "movie" so i want it to be in the "movies" collection.
I can make an array of users in the "movies" collection, and add the UID there so i can check if the current user voted or not, maybe this is the correct way ?
I don't think more than 1000 people are gonna use the web app i'm making, it's for a very specific non popular thing, so i think it should be fine even if there are 20k people in the next 5 years.?
"In a state I like to call 'getting uninstalled'" 🤣🤣🤣
What's the best way to store say I am expecting to store 10m keys. The workflow goes like this If I encounter a key in my app, I check firestore if that key exists, if it does I do some thing if not I put the key in the firestore. I don't really care for the value of the key if there is provision to delete the key or else let the value be true/false, such that I can use false if I wish to delete the key from firestore. I need any of key insertion/lookup/delete in O(1) or O(logN) .
"Wrap it up Charles Dickens, you are aren't getting paid by the word."
-Gene, Bob's Burgers
Also, still a great tutorial 3 years later.
Anyone here who are struggling to get firebase working in angular 10? getting this error
This likely means that the library (@angular/fire/firestore) which declares AngularFirestore has not been processed correctly by ngcc, or is not compatible with Angular Ivy. Check if a newer version of the library is available, and update if so. Also consider checking with the library's authors to see if the library is expected to be compatible with Ivy.
8:44 Can you please point to an example of how such an index should look like?
Great!! and how can I do a query like "get all characters from each book into 'dickens_book'" I mean get all subcollections 'characters' from all dickens's books in orther to save them into a array after that?
can this 20k document field limit be bypassed if we can turn off indexing?
Very helpful vid! But only one write per document per second... WAT
So... is the Charles Dickens app market booming yet?
Ok, wheres the video about collectionGroup?
How can I prevent indexing?
Should I store a serialized JSON object (as firestore String)?
You're great but to keep up I have to slow down the playback speed :D
me too lol... he speaks to fast.
does this mean that if a document only contains one array, the size of the array is also limited to 20,000 elements (like maps) as well?
Problem with subtitles from 6:23 :(
use google voice translate
Do you have videos for firebase 9
I want to list the unread "news" if the "viewed" array within the news documents doesn't contain the user id...
Any ideas on how to implement the logic "notarrayContains" ? I guess firestore does not support it?
Thanks for any help!
For those who are wondering how to achieve this:
groups.google.com/forum/?#!msg/google-cloud-firestore-discuss/wXn4nd2RBIc/jsWhxy3FBgAJ
If I want to store user's tenats, should I create an array of references?
#askfirebase
Are arrays ordered as they are sent?
Suppose I write [ b , a , c]
are they stored in the same order or [ a , b , c ] ?
If array_contains is like map field === true, why can we only query on 1 value at a time, and not multiple ?
So expensive cloud firestore 😩
8:04 Top-level Collections
What if I have situation where ,logically speaking, I should have one document with few subcolletions and each subcollection have documents with many subcolletions. And i always wanna show whole document or just root (top level without any subcolletions) But i know that many users are gonna write and change that document very often, so should I split it to top level collections to avoid write failing ?
If I query a collection with 40 documents, will I get charged for each of those documents reads?
I can't find any documentation to allow me to update my array of maps in Firestore from flutter. Any ideas on how to find this?
Is it possible to write security rules for arrays or is a sub collection the only way?
A rule applies to an entire document (ie ALL its fields including arrays) so if you needed to protect data differently to the rest of the document then I would put it in a sub collection.
I have a doubt regarding deleting an array of data in Arrays. The Structure looks like
image: [
0 :
imageurl : ".........",
isliked : "true",
1:
imageurl : ".........",
isliked : "false",
]
now I'd like to delete index 1 how can I delete using the Cloud Firestore ArrayRemove method or is there anything else to remove based on Index position. Thanks & Regards :)
@Firebase Where is the documentation of the new array features? I couldn't find in here -> firebase.google.com/docs/firestore/manage-data/delete-data
hey but how can I get an array from the collection I did not find any documentation in a firebase site??
Hey,Please make a video on how to maintain data consistency across all collection and documents.I found write batch for this problem but there is limit of only 500 per one write batch. What if i want to write more than 500 at once.
NoSQL implies you, the dev, handle manually data consistency at write time. Once you edit a record, you have to apply changes on all collections where it is referenced. That said, you really don't want to use foreign keys, which is an artefact of SQL. Generally; you separate stuff and you end up with top level collections such : /users/{userId} describing your user, or /friends/{userId} describing the friends of your user, or /messages/{userId} describing messages of your user.
Yeah cool thanks.. But still there is a limit of 500 write batch at once what if there are more than 500 duplicate Data and i want to update them, i can only update 500 at once not more than that.
I found that splitting my triggers across multiple functions helped a lot with that limit. Of course if you still have such large batches then write all updates to an array then step through the array to create separate batches of 500 each and then use Promise.all() to run them.
How do they do these video edits from the future?
If a document can have just 20000 fields max, then its a huge problem to something like comments and replies. A document for comments can have fields for each comment with subfields for comment meta data(eg. time,comment text , comment by,etc). Further for each comment field(which is in the form of a map), can have another map for replies. This would easily and in no time consume those 20000 fields.
If we use sub collections for each document or even each reply then it would just cost huge for reads as the number of comments and replies grow.
Please tell me if i am missing something and the solution for this problem as till now firestore was my favorite, but now i feel i need to switch to something else :(
Also posted as a question on stackoverflow.com/q/62453703, where Doug already responded. Unless you have a different question here, I'd recommend following up on Stack Overflow if you have further questions.
hey, how would I query if I had a collection called videos tat contain the video url etc and another collection called users tat contains the username which needs to be displayed during the video play.
Is it possible to update a map in an array ?