Fun fact: I referenced a video of yours about SQLite for a NotebookLM generated podcast. The "host's" now call SQLite "Squeel Lite" now. Thanks for nothing pal! :)
Both use pages. If you have 10 files and you want to use them, your app must use fopen() ten times. But if those same files are stored in a Sqlite database then we only have to use fopen() one time. Thats the difference.
@@martijnb3381that's not all, it also depends on how one is updating the JSON file, if a program is just dumping/stringifying and writing the whole data on each update, that's also a issue, especially if said data is tens or hundreds of MB.
Exactly what the creator of SQLite said. "SQLite was not created to go up against databases like Postgresql or MySql, it was created to go up against flat file databases."
We should all endeavor to re-invent the wheel once. If for nothing more than to get it out of your system. But the key is to do that BEFORE people start paying you to code.
@@seanwoods647 Or, do it in the downtime when no major features is being planned. When code has gone to production you get lots of use cases that _should_ be turned into Tests.
I use SQLite for my beginner projects whenever I start a new programming language, typically a TODO list (yes, technically reinventing wheel, no I dont care that there's alot, its for learning, people should reinvent the wheel to learn), and its amazing
7:30 When you hold a non-async safe lock across an async boundary, the executor might suspend the task for an unknown amount of time. During that time another task that waits on the same mutex might be woken up and then wait until the first task lets go of the mutex, which will block the executor from continuing to execute the first task, which leads to a deadlock where all threads of the executor are all blocked by the mutex.
Yep. And async mutex explicitly exist to avoid this dead lock. For whatever reason they didn't use it. Looks like an application bug and not a rust bug.
@@justanothercomment416 I mean that it was an application bug was never in question. And using a non-async lock in async code, *actually* can have its advantages. async locks work by keeping a linked list of the task wakers that are waiting on it, which has a certain overhead. So if you know you *never* keep a lock over an async-await point, it might actually worth using it. (Especially in some low-level code, for example when implementing a future manually.)
@@justanothercomment416 It's actually officially recommended in async Rust book and I think in a Jon Gjenset video about async, to use *sync Mutexes* across small critical sections, *as long as you don't straddle an async boundary* . Of course, that turned out to be harder than it looks.
@@comradepeter87 Interesting. Because I looked before I even commented and found async mutexes to be the defacto recommendation. It would appear some of those with "authority" don't really understand what they are saying. Because the async implementation explicitly exists to avoid all of this nonesense.
@@justanothercomment416 I.. highly doubt that. The "official", or as you rightly called it - "authoritative" recommendations do explicitly point out these potential problems. But the thing is, async mutexes are very slow. And a lot of the times, the critical section is really small and doesn't involve an async operation, maybe like pushing into a HashMap, or updating a counter, etc. It did irk me too the first time I heard it, as it can create the problems precisely mentioned here. However, it is a risk people are willing to take, and as we saw sometimes it might not pay off.
Rawdogging sync rust is indeed the way the lord intended it. But i am using Embassy, which is async and i am loving it. Calling await on a physical pin that resolves when there is physically 3.3v on it is wild.
@@marwan7614 yup. that's what they are under the hood. but you don't have to write anything by hand. just throw in an await and the computer does it's thang.
@@schachmatsch4790 no shit, but the fact that he says it so clearly and boldly is a lot different than others that just have a small notice somewhere most people won't directly see it
@@p0xygen Except for the fact that he hedges. "Turso does pay me" and "This article, I guess, you would definitely be able to call this thing sponsored." What thing? Is he saying it just because Turso pays him for something else? What thing is he talking about, this youtube video? If Turso is paying him to make the video, just say so.
In .NET, calling Wait (sync) on an async function can also result in a deadlock under certain scenarios and has to be avoided, so this isn't just a RUST thing.
Don't run aggregate jobs across the databases, run it across your event log. Give every user a mutable DB that stores the live data, but pool all the user events for aggregation.
I work with multi-tenant databases. The issues usually come from not balancing the load correctly. In our testing we found pretty much every version of standard squeal postgres Microsoft whatever to be faster than squeal light
SQLite is used for embedded systems and stand alone applications. If you find yourself needing to load balance servers you’re using the wrong tool. Next try to use resident Postgres for your cell phone app. Different tools for different jobs.
@@mattymattffs Yeah but turso doing it 1 database per user and they put those on sqlite on s3 if not used I think multi tenant isnt quite the good word here because they essentialy "not" multi tenant in a way most multi tenant works
@@hbobenicio It's a mutex which understands Rust's async wait and behaviors correctly for these types of use. Whereas a normal mutex blocks execution as it's becoming recursive, which isn't what you would want for an async operation and its scheduler in the first place. The "deadlock" is the block as requested by the mutex. While making some assumptions as the full context was not provided, it appears this was not a Rust bug but a user bug from using the incorrect mutex for purpose.
The way you do aggregates (or rather a way), is to use an external cache, something like a redis or whatnot, to save the per-user metrics you want to aggregate, and then work on the cache when it comes to finalizing the numbers.
19:33 SQLite is a regular filesystem file. There is no real usable Async filesystem kernel routine. All async fs stuff is run in a thread pool under the hood and appear to be async using fake syntax sugar. In all languages.
Did they reinvent the wheel? Or did they make of their own wheel so that they could have the wheel that they need? I just feel like this saying inappropriately used sometimes.
Semantics. They reinvented the wheel. The only time it isn't is when it innovates because a solves notable problems or improves existing standards. Reinvented for your niche is still reinventing.
@@audiocorps2334is it "reinventing the wheel" when what you did was make the wheel skinnier and taller for your specific use-case, or is it just "making the wheel I need"?
@@Kane0123 it is probably the most prolific piece of software of all time, you find it everywhere from your iphone to airplanes to missiles to hospitals. a major vulnerability in sqlite would shutdown a huge part of the world's economy.
It's always been good for client side storage in apps, all these people trying to shoehorn it into servers and hacking it into a server-side database are misguided. That's just not it's purpose.
28:00 I guess it depends on your definition of "cold start". I assumed it would have meant that every database has all the important bits in the filesystem cache in any situation. If it actually meant "not having to boot a new VM", sure, that's much much easier thing to accomplish if you don't need VM level separation for security purposes. And considering the amount of hardware security bugs related to VM escape vulnerabilities, I would rather bet on well written server software instead of VM containment.
i don't get it. They are creating a service for creating and accessing SQLite database files? why? anyone can easily create their own SQLite database files on their own computer or server.
The best way to ensure no problems is by user journey testing. For example create a todo item, search for it, edit the todo, search again, close todo, search again, ensure it’s removed from the dashboard, then delete. Then everything needs instrumented to catch the random user issues in prod so the users do not have to contact support which they rarely do. DST sounds interesting
I once got bamboozled doing that kind of research for myself. I wrote my crud app building my own scheduler using rusts mio non blocking event loops for any hardware io and thread pools with sync channels for work dispatch. It was very lightweight on ram and rather fast. But when spamming it with simulated ddos attacks, it hung up and didnt answer any more requests at some point. I gave up and rewrote the entire thing using tokio. And the problems went away! Just found out a week later that my experience was most likely caused by a bug in rustls crate that i used in the first version that was not present in the tokio-rustls crate i used for the second. This bug was so bad, it was reported as a CVE because it hung up some place in the tls handshake that could be exploited. Still am wondering about speed comparisons of both implementations with the bug in rustls being fixed by now. But didnt get around to rewriting it again yet :D
@7:51, a lot of thread synchronization and locks utilize thread local storage. These are not safe in async code since this code and the release of the locks might not run on the same thread. C# is introducing async task local storage to create async safe synchronization and locks.
We do single-tenant DBs at work. Orchestrating them is easy by having a single system/admin db where "pointer" records about each of the account exist. All the system-wide stuff also goes there of course
@21:22 The issue is the "move" keyword: Everything referenced in this block will be moved in! I think creating a similar "clone" and "borrow" block would make sense. But you would loose the explicit `.clone` and `&` in those. For what it is worth, the move block semantics is not that bad and it forces you to be explicit and correct. But yeah... so many people raise this issue.
1:00 I think even companies can benefit from reinventing the wheel. Depending on what your company or project within that company does, all the various "wheels" might only almost work for you. Libraries are rarely general enough to fit all situations, and having a bunch of corner cases where you need to use inevitable hacks to work with a library/tool builds up into tech debt over time. Specialized projects/libraries can also turn into debt, but that debt is generally very stable and not too painful.
It would be cool to implement this testing strategy on a very simple project with different branches or components for triggering different bugs just to learn/teach.
i always use it in my small project where i willnt handle many data since last month, my reason for switching from postgres to sqlite for my personal project is because i dont have to set up postgress in the server when i deploy it to my vps. i just run the binary (i embed the sqlite in the binary + the frontend) and everything is done
Yup, if you don't actually need an enterprise-grade db and/or your data set is small and/or your architecture does not need a separate database, SQLite is just perfect #Chef'sKiss
I don't think the deadlock is Rust specific, except in the matter that Rusts Mutex is non-entrant, where-as a reentrant Mutex might not have caused the deadlock (but that might cause other problems!). Basically thread A locks the mutex and calls some_sync_function. Then some_sync_function does async stuff that causes the thread A to yield to the executor to run another task, via thread A. Thread A then get another request and runs the initial code again which tries to lock the Mutex. But since Rust's Mutex is not re-entrant thread A is now deadlocked with itself and the Mutex is perhaps never released. All other threads handling request now also deadlocks on the perma-locked Mutex.
7:23 "Why can't you hold a mutex across an async function?" You can, but the problem is when you .await. There's a lot underlying rust's future model here, but basically the future will *always* return pending or will just straight up block the thread forever, depending on if you used a mutex that returns a future (ala, an async mutex from tokio or smol) or one that just goes and contacts the operating system for sync (std or crossbeam). The way around this comes down to just not mixing mutexes, or not doing recursive locking.
I am impressed by things people are creating and I am jealous. The first 10 years of my career I spent building CRUD and video streaming apps (hooking third party service). Just 3 years ago, I deep dived into more interesting topics.
The issue with async + mutex is that async is all about interweaving work on a thread - the thread "parks" the work and goes and does something else whilst waiting for the async task to return - do if the mutex is implemented at thread level (common) it blocks the entire thread and all async stuff happening on said thread. Easy to deadlock. Also if you understand multithreaded code in your language / environment / os etc you'll get right during analysis. I mean you do have to understand how async works under the covers before you go writing concurrent code, right? Or is that just me?
Can you mention that if you are paid to review/read an article whether that also has impact on your opinion on it? Because I automatically assume bias when something is sponsored
sqlite will be the db of choice for my current hobby project getting used to rust. If it just had a good timeseries module so I could throw influxdb out of the stack :D
why query directly on the database at a lightening speed with SQL, when you can spend time writing python code to connect to the same database just to join and aggregate tables with inefficient python code.
Re-inventing the wheel can be seen in different ways. Dogmatic DRY people say it all the time, and they ignore the learning opportunity. People who love frameworking instead of just writing code re-invent the wheel in a negative way because they often make frameworks that ignore or even prevent using what built-in features a technology has. (Like J.S. frameworks that don't let you use html5 goodness)
Antithesis = An-ti-thu-siss. Prime reading skill issues aside /s, Fantastic video and highlighted article. SQLite is great and Turso helps emphasise this here. Anything involving Tigerstyle with DST just vibes with me these days. My favourite way to get shit done both in professional work and personal projects especially.
The unit tests I write are just straightforward testing practices that you'd find in any good book, and they catch all sorts of things I would not have thought about or predicted. I rely pretty much exclusively on just run-of-the mill testing practices, unit tests written in a basic unit test framework (a-la xUnit or whatever), and I haven't had ay code defects in anything I've released in literally years. I also write a lot of runtime asserts checking before/after contracts. Of course: I've seen other people's tests and they most certainly fit this "only catch what you predicted" criticism. In fact more often they don't even catch what you would predict; more often than not if you look at the test it's actually testing nothing at all: no asserts that could fail, or everything meaningful is mocked out. That's not because "testing is limited", that's because people limited: they are not trained or competent at testing, and the organization largely tries to indoctrinate them to be uncritical and unthinking which goes against the ability to do testing well. Do you think calling it "simulation testing" is going to change anything in the long run? If that gets popular, the majority will just do that terribly too.
"everything meaningful is mocked out" dependencies should be mocked or you will be testing multiple classes at the same time, you should have separate test for the depedency that test it in isolation.
what's the thing with VMs. When did wasting resources for subpar performance for isolation whatever reasons become a thing. Then they have to spin up/down virtual machines, on a real hardware.
For the example shown at 21:47, while it would be nice to have some shorthand, you can at least not pollute naming with inner or inner_something by doing the clone within task::spawn but before the async move {} let a = Rc::new(something); handle = tokio::task::spawn({ let a = a.to_owned(); async move { inner_usage(a); } }); other_usage(a)
I've always been iffy about serverless. I just hate the idea of building something on serverless then the platform decides to raise rates and I am sort of stuck because switching providers is too costly in time/money. I've always preferred running on stuff that can easily be self hosted or moved to another provider without any issue and because of that self hosting serverless just seems pointless. Serverless definitely has it's use cases but I have yet to find a use case for my stuff that can't easily be done another way and without the fear of my bill exploding.
IMO serwverless is for projects that have no budged to expend in infrastructure, serverless is really expensive compared with cloud VMs and PaaS oferings like Azure App Service or AWS Lightsail. also migrating away from serverless is most of the time, rewriting the entire app
Async sucks. Zig killed async for solid reasons. Google killed io_uring. You use Tokio for direct IO, rayon for bulk parallel computation. But um, all kinds of issues. Also stare at Concurrency for LLVM. If they can salvage async, would be amazing. Cancel blockage, function pointers, measuring memory, hidden control flow, coloring. Ya, I wanna see. There's no roll your own to reduce syscalls. It's a nightmare. He would be the one to fix it though.
Per user db is bullshit in 99% cases, but there might be 1% cases where it makes sense. Like some online photoshop, or other offline-like tools used in online.
I haven't used multi tenancy in production yet because too much overhead maybe I haven't implemented the correct pattern but I would appreciate really super simple multi tenancy I think I came across the same issues as these dudes
Greetings from the Netherlands, the Primeagen. I am not sure if you noticed the acrostic in the section "Appendix: Why not Zig?" Look at the boldly printed first letter of each sentence.
The SQLite project does not accept patches from people who have not submitted an affidavit dedicating their contribution into the public domain. This means that Turso does not want to submit an affidavit dedicating their contributions into the public domain.
Or it means that Turso might want to implement features that are specific to their use case and are unlikely to be accepted for merge into SQLite. Also, fyi, Turso devs have actually up streamed changes back to SQLite.
No cold start can be handled the same way android handles its VMs. You basically pre-fork the initialized base VM (zygot in Android speak IIRC). This way, you're always connecting to a pre-warmed image rather than cold starting on each need.
I don't like a lot of things about k8s or docker... or ansible/tower etc. reinventing all the stuff as well... including server, security, auth, rpc, protocols etc etc etc. Why. Been trying to bring sim testing into an app layer 4evar... but it's never the app that's the issue...
Ansible and Salt are great tools to ensure identical system config if one doesn't use Nix. Those and Terraform/OpenTofu means you truly can rebuild your whole infrastructure quickly when disaster happens.
The solution was that time 20 yesrs ago use someyhing faster than os thread switch that time that was directx timers. so we had like 240 voip streams + control sent and recieved 30ms cunks while os alowed only 1ms delta. Mutex in hardware interupt callback caused trouble.
I came here for Sqlite but the presentation just mentioned it and then took a hard turn to Rust and deadlocks.
Fun fact: I referenced a video of yours about SQLite for a NotebookLM generated podcast. The "host's" now call SQLite "Squeel Lite" now. Thanks for nothing pal! :)
that's crazily hilarious 🤣🤣
😂
try not to train ure models on inferior data sources
I used to say it ironically. Now I say it "squeel lite" without even thinking about it
Once you hear squeel, you can never go back. It's just too catchy and correct.
Sqlite is godly. I started using it instead of json files for small projects. Somehow it's faster for tiny datasets. Didn't expect that.
It’s faster than the file system, check their blog
@@paladin9876 yeah cause file system works on pages, while sqlite directly uses disk.
Both use pages. If you have 10 files and you want to use them, your app must use fopen() ten times. But if those same files are stored in a Sqlite database then we only have to use fopen() one time. Thats the difference.
@@martijnb3381that's not all, it also depends on how one is updating the JSON file, if a program is just dumping/stringifying and writing the whole data on each update, that's also a issue, especially if said data is tens or hundreds of MB.
Exactly what the creator of SQLite said. "SQLite was not created to go up against databases like Postgresql or MySql, it was created to go up against flat file databases."
1:07 Reinventing the wheel is a fantastic way to understand the _wheel_ more broadly.
We should all endeavor to re-invent the wheel once. If for nothing more than to get it out of your system. But the key is to do that BEFORE people start paying you to code.
@@seanwoods647 Or, do it in the downtime when no major features is being planned. When code has gone to production you get lots of use cases that _should_ be turned into Tests.
"Blame the tools."
I literally LOL'd. Perfect delivery.
To be fair, sometimes the tools are indeed buggy 😅
i never would've thought that was funny ever, thank you for making it funny that you find it funny
Imagine when Prime discovers formal methods and starts writing specifications
He'll finally write Haskell..
Prime and writing D and Ada? No way
He might even learn to spell TLA+
I use SQLite for my beginner projects whenever I start a new programming language, typically a TODO list (yes, technically reinventing wheel, no I dont care that there's alot, its for learning, people should reinvent the wheel to learn), and its amazing
Bro same. Together we can flood the market and monopolize shitty to do apps.
The expert system I maintain (IRM) is actually one of the original applications that Sqlite was developed for.
7:30 When you hold a non-async safe lock across an async boundary, the executor might suspend the task for an unknown amount of time.
During that time another task that waits on the same mutex might be woken up and then wait until the first task lets go of the mutex, which will block the executor from continuing to execute the first task, which leads to a deadlock where all threads of the executor are all blocked by the mutex.
Yep. And async mutex explicitly exist to avoid this dead lock. For whatever reason they didn't use it. Looks like an application bug and not a rust bug.
@@justanothercomment416 I mean that it was an application bug was never in question.
And using a non-async lock in async code, *actually* can have its advantages.
async locks work by keeping a linked list of the task wakers that are waiting on it, which has a certain overhead.
So if you know you *never* keep a lock over an async-await point, it might actually worth using it.
(Especially in some low-level code, for example when implementing a future manually.)
@@justanothercomment416 It's actually officially recommended in async Rust book and I think in a Jon Gjenset video about async, to use *sync Mutexes* across small critical sections, *as long as you don't straddle an async boundary* . Of course, that turned out to be harder than it looks.
@@comradepeter87 Interesting. Because I looked before I even commented and found async mutexes to be the defacto recommendation. It would appear some of those with "authority" don't really understand what they are saying. Because the async implementation explicitly exists to avoid all of this nonesense.
@@justanothercomment416 I.. highly doubt that. The "official", or as you rightly called it - "authoritative" recommendations do explicitly point out these potential problems. But the thing is, async mutexes are very slow. And a lot of the times, the critical section is really small and doesn't involve an async operation, maybe like pushing into a HashMap, or updating a counter, etc. It did irk me too the first time I heard it, as it can create the problems precisely mentioned here. However, it is a risk people are willing to take, and as we saw sometimes it might not pay off.
Rawdogging sync rust is indeed the way the lord intended it. But i am using Embassy, which is async and i am loving it. Calling await on a physical pin that resolves when there is physically 3.3v on it is wild.
hardware interrupts ?
@@marwan7614 yup. that's what they are under the hood. but you don't have to write anything by hand. just throw in an await and the computer does it's thang.
🔥 love the sponsorship Acknowledgement straight out of the gate.
you do realize it is required by law right?
@@schachmatsch4790 no shit, but the fact that he says it so clearly and boldly is a lot different than others that just have a small notice somewhere most people won't directly see it
What's it with you guys and sponsorship.
If a video is sponsored and the creator lies about the content; I stop watching the creator. simple
@@schachmatsch4790 I'm not familiar with the law. Platform thing or something else?
@@p0xygen Except for the fact that he hedges. "Turso does pay me" and "This article, I guess, you would definitely be able to call this thing sponsored." What thing? Is he saying it just because Turso pays him for something else? What thing is he talking about, this youtube video? If Turso is paying him to make the video, just say so.
Sqlite is qualified for Aerospace.
Is that the new “military grade” meaningless label for things now. The “milspec” of tech?
Boeing is qualified for Aerospace
@@JohnSmith-op7lsDO-178B is the certification. I think its not a buzz word like "milspec" 😊
@@allesarfintYou win this comment section
So you're telling me Sqlite is missing a few bolts...
Petition to actually change it to squeel.
Imagine MySqueel or PostGresqueel
you can't change it to what it is
literally impossible, comrade
In .NET, calling Wait (sync) on an async function can also result in a deadlock under certain scenarios and has to be avoided, so this isn't just a RUST thing.
yep. i was thinking the same
Can you give a example or is it a 1 in one Million thing?
@@schmidt-5099 I can't post links but search for article "Understanding Async, Avoiding Deadlocks in C#"
Search for article "Understanding Async, Avoiding Deadlocks in C#"
Understanding Async, Avoiding Deadlocks in C# article explains this more in-depth
DST is the acronym for STD in my native language. Listening to this video was an experience.
I'm not sure if Daylight Saving Time is that in my timezone.
In the Star Wars Jedi Academy community it refers to Dark Side Tools which is a bunch of hacks like wallhack, aimbot etc which is quite notorious lol
Don't run aggregate jobs across the databases, run it across your event log. Give every user a mutable DB that stores the live data, but pool all the user events for aggregation.
I work with multi-tenant databases. The issues usually come from not balancing the load correctly. In our testing we found pretty much every version of standard squeal postgres Microsoft whatever to be faster than squeal light
SQLite is used for embedded systems and stand alone applications. If you find yourself needing to load balance servers you’re using the wrong tool. Next try to use resident Postgres for your cell phone app. Different tools for different jobs.
@@ElSantoLuchador I don't think you understood. Turso is saying they are doing multi tenant SQL lite databases. I'm saying that's dumb
@@mattymattffs to be honest,I didn't get waht you mean in the first comment too. but I enterily agree with you
@@mattymattffs Yeah but turso doing it 1 database per user and they put those on sqlite on s3 if not used
I think multi tenant isnt quite the good word here because they essentialy "not" multi tenant in a way most multi tenant works
Not entirely clear, but for that mutex, it should be an async mutex. If using the standard mutex described bug is expected.
wth is an async mutex?
@@hbobenicio It's a mutex which understands Rust's async wait and behaviors correctly for these types of use. Whereas a normal mutex blocks execution as it's becoming recursive, which isn't what you would want for an async operation and its scheduler in the first place. The "deadlock" is the block as requested by the mutex.
While making some assumptions as the full context was not provided, it appears this was not a Rust bug but a user bug from using the incorrect mutex for purpose.
The way you do aggregates (or rather a way), is to use an external cache, something like a redis or whatnot, to save the per-user metrics you want to aggregate, and then work on the cache when it comes to finalizing the numbers.
I love SQLite. It’s so versatile and fast as hell.
The wheel has been re-invented probably more than any other part on your car
Fun fact: I have the job I have after striking up a conversation with Richard Hipp while in line at a conference. The power of the right T-shirt
i love that every point in the conclusion of why spells out "borrow checker" aka thats the whole reason why.
19:33 SQLite is a regular filesystem file. There is no real usable Async filesystem kernel routine. All async fs stuff is run in a thread pool under the hood and appear to be async using fake syntax sugar. In all languages.
NT Native API comes looking 😉
Sqlite is best for read heavy usage. but for write heavy usage, we should prefer other databases.
Did they reinvent the wheel?
Or did they make of their own wheel so that they could have the wheel that they need?
I just feel like this saying inappropriately used sometimes.
Semantics. They reinvented the wheel. The only time it isn't is when it innovates because a solves notable problems or improves existing standards.
Reinvented for your niche is still reinventing.
tubeless tires were a reinvention of the wheel
@@audiocorps2334is it "reinventing the wheel" when what you did was make the wheel skinnier and taller for your specific use-case, or is it just "making the wheel I need"?
Sqlite has always been good
I feel like a lot of us were just sleeping on it...
it's pretty much the only option for mobile
@@Kane0123 it is probably the most prolific piece of software of all time, you find it everywhere from your iphone to airplanes to missiles to hospitals. a major vulnerability in sqlite would shutdown a huge part of the world's economy.
It's always been good for client side storage in apps, all these people trying to shoehorn it into servers and hacking it into a server-side database are misguided. That's just not it's purpose.
@@justinsmith3981in flutter i use isar, nosql local database, it's FAST
28:00 I guess it depends on your definition of "cold start". I assumed it would have meant that every database has all the important bits in the filesystem cache in any situation. If it actually meant "not having to boot a new VM", sure, that's much much easier thing to accomplish if you don't need VM level separation for security purposes.
And considering the amount of hardware security bugs related to VM escape vulnerabilities, I would rather bet on well written server software instead of VM containment.
i don't get it. They are creating a service for creating and accessing SQLite database files? why? anyone can easily create their own SQLite database files on their own computer or server.
The best way to ensure no problems is by user journey testing. For example create a todo item, search for it, edit the todo, search again, close todo, search again, ensure it’s removed from the dashboard, then delete. Then everything needs instrumented to catch the random user issues in prod so the users do not have to contact support which they rarely do. DST sounds interesting
Prime you should get Joran on stream to ask for advice with your simulation testing questions! It would be a really fun conversation
I once got bamboozled doing that kind of research for myself. I wrote my crud app building my own scheduler using rusts mio non blocking event loops for any hardware io and thread pools with sync channels for work dispatch.
It was very lightweight on ram and rather fast. But when spamming it with simulated ddos attacks, it hung up and didnt answer any more requests at some point.
I gave up and rewrote the entire thing using tokio. And the problems went away!
Just found out a week later that my experience was most likely caused by a bug in rustls crate that i used in the first version that was not present in the tokio-rustls crate i used for the second. This bug was so bad, it was reported as a CVE because it hung up some place in the tls handshake that could be exploited.
Still am wondering about speed comparisons of both implementations with the bug in rustls being fixed by now.
But didnt get around to rewriting it again yet :D
I'm building a project that has a SQlite DB per tenant, it's super stable and easy to do.
how many tenant? how you handle table/column change?
@7:51, a lot of thread synchronization and locks utilize thread local storage. These are not safe in async code since this code and the release of the locks might not run on the same thread. C# is introducing async task local storage to create async safe synchronization and locks.
3:24 what is sequel lite?
We do single-tenant DBs at work. Orchestrating them is easy by having a single system/admin db where "pointer" records about each of the account exist. All the system-wide stuff also goes there of course
Does anyone where you can find the presentation that prime mentions around 9:35?
First time see Prime reading an articles positively.
@21:22 The issue is the "move" keyword: Everything referenced in this block will be moved in!
I think creating a similar "clone" and "borrow" block would make sense. But you would loose the explicit `.clone` and `&` in those.
For what it is worth, the move block semantics is not that bad and it forces you to be explicit and correct. But yeah... so many people raise this issue.
My only complaint about SQLite is i wish it would enforce column types
Strict mode. It’s in the docs
1:00 I think even companies can benefit from reinventing the wheel.
Depending on what your company or project within that company does, all the various "wheels" might only almost work for you. Libraries are rarely general enough to fit all situations, and having a bunch of corner cases where you need to use inevitable hacks to work with a library/tool builds up into tech debt over time.
Specialized projects/libraries can also turn into debt, but that debt is generally very stable and not too painful.
2:17 someone will hear the file based database and say aha I know my excel skills are not wasted.
It would be cool to implement this testing strategy on a very simple project with different branches or components for triggering different bugs just to learn/teach.
i always use it in my small project where i willnt handle many data since last month, my reason for switching from postgres to sqlite for my personal project is because i dont have to set up postgress in the server when i deploy it to my vps. i just run the binary (i embed the sqlite in the binary + the frontend) and everything is done
Sqlite just works 😊
Yup, if you don't actually need an enterprise-grade db and/or your data set is small and/or your architecture does not need a separate database, SQLite is just perfect #Chef'sKiss
I don't think the deadlock is Rust specific, except in the matter that Rusts Mutex is non-entrant, where-as a reentrant Mutex might not have caused the deadlock (but that might cause other problems!).
Basically thread A locks the mutex and calls some_sync_function. Then some_sync_function does async stuff that causes the thread A to yield to the executor to run another task, via thread A. Thread A then get another request and runs the initial code again which tries to lock the Mutex. But since Rust's Mutex is not re-entrant thread A is now deadlocked with itself and the Mutex is perhaps never released. All other threads handling request now also deadlocks on the perma-locked Mutex.
7:23 "Why can't you hold a mutex across an async function?"
You can, but the problem is when you .await. There's a lot underlying rust's future model here, but basically the future will *always* return pending or will just straight up block the thread forever, depending on if you used a mutex that returns a future (ala, an async mutex from tokio or smol) or one that just goes and contacts the operating system for sync (std or crossbeam). The way around this comes down to just not mixing mutexes, or not doing recursive locking.
I am impressed by things people are creating and I am jealous. The first 10 years of my career I spent building CRUD and video streaming apps (hooking third party service). Just 3 years ago, I deep dived into more interesting topics.
I totally knew you were going to say “shill-o-gen”!
so funny to see an idea you thought of on the throne the day before being discussed in a youtube video, I love it
Why do they need to create all 5000 db files in one call? MQ that thing and have "eventual" creation. Still fast but no threadaches?
The issue with async + mutex is that async is all about interweaving work on a thread - the thread "parks" the work and goes and does something else whilst waiting for the async task to return - do if the mutex is implemented at thread level (common) it blocks the entire thread and all async stuff happening on said thread. Easy to deadlock. Also if you understand multithreaded code in your language / environment / os etc you'll get right during analysis. I mean you do have to understand how async works under the covers before you go writing concurrent code, right? Or is that just me?
Can you mention that if you are paid to review/read an article whether that also has impact on your opinion on it? Because I automatically assume bias when something is sponsored
sqlite will be the db of choice for my current hobby project getting used to rust.
If it just had a good timeseries module so I could throw influxdb out of the stack :D
@@Th1200 sqlite3 has a robust extension api. Write one
why query directly on the database at a lightening speed with SQL, when you can spend time writing python code to connect to the same database just to join and aggregate tables with inefficient python code.
Re-inventing the wheel can be seen in different ways. Dogmatic DRY people say it all the time, and they ignore the learning opportunity.
People who love frameworking instead of just writing code re-invent the wheel in a negative way because they often make frameworks that ignore or even prevent using what built-in features a technology has. (Like J.S. frameworks that don't let you use html5 goodness)
W editor ❤️
I did a multi-tenant encrypted at rest database out of SQLite3 16 years ago.
I quite literally have the tigerstyle video in the next tab. Didn't realize that was that.
Antithesis = An-ti-thu-siss.
Prime reading skill issues aside /s,
Fantastic video and highlighted article. SQLite is great and Turso helps emphasise this here.
Anything involving Tigerstyle with DST just vibes with me these days. My favourite way to get shit done both in professional work and personal projects especially.
0:00 Did he say squirrel it? Like making something become a squirrel?
This is what you call a solution in search of a problem kids.
The unit tests I write are just straightforward testing practices that you'd find in any good book, and they catch all sorts of things I would not have thought about or predicted. I rely pretty much exclusively on just run-of-the mill testing practices, unit tests written in a basic unit test framework (a-la xUnit or whatever), and I haven't had ay code defects in anything I've released in literally years. I also write a lot of runtime asserts checking before/after contracts.
Of course: I've seen other people's tests and they most certainly fit this "only catch what you predicted" criticism. In fact more often they don't even catch what you would predict; more often than not if you look at the test it's actually testing nothing at all: no asserts that could fail, or everything meaningful is mocked out. That's not because "testing is limited", that's because people limited: they are not trained or competent at testing, and the organization largely tries to indoctrinate them to be uncritical and unthinking which goes against the ability to do testing well. Do you think calling it "simulation testing" is going to change anything in the long run? If that gets popular, the majority will just do that terribly too.
"everything meaningful is mocked out" dependencies should be mocked or you will be testing multiple classes at the same time, you should have separate test for the depedency that test it in isolation.
This just unlocked new pathways in my mind
Sync event loop in Rust sounds really interesting, would love to see it along with a zero cold start serverless server.
what's the thing with VMs. When did wasting resources for subpar performance for isolation whatever reasons become a thing. Then they have to spin up/down virtual machines, on a real hardware.
SurrealDB already has multi tenancy/db per user
Woah the word multi tenancy turned into a link
@@kv4648 for me it’s not, but i am seeing a bunch of other links. Mindblowing. TH-cam showing those based uniquely per user? Interesting…
1:31 the r-word is "rust"
what is the most performance open source database now?
I use it all the time for personal projects. It’s amazing
For the example shown at 21:47, while it would be nice to have some shorthand, you can at least not pollute naming with inner or inner_something by doing the clone within task::spawn but before the async move {}
let a = Rc::new(something);
handle = tokio::task::spawn({
let a = a.to_owned();
async move { inner_usage(a); }
});
other_usage(a)
I've always been iffy about serverless. I just hate the idea of building something on serverless then the platform decides to raise rates and I am sort of stuck because switching providers is too costly in time/money. I've always preferred running on stuff that can easily be self hosted or moved to another provider without any issue and because of that self hosting serverless just seems pointless.
Serverless definitely has it's use cases but I have yet to find a use case for my stuff that can't easily be done another way and without the fear of my bill exploding.
IMO serwverless is for projects that have no budged to expend in infrastructure, serverless is really expensive compared with cloud VMs and PaaS oferings like Azure App Service or AWS Lightsail.
also migrating away from serverless is most of the time, rewriting the entire app
It's deeply silly to me, just make people download it at that point
It will be fun to watch all developers who migrated their projects to SQLite when the ops guys ask "And how are we supposed to do backups now?!"
What do you think about DuckDB
Can you make a video on how you implemented the seeded simulation testing in your go project?
Async sucks. Zig killed async for solid reasons. Google killed io_uring. You use Tokio for direct IO, rayon for bulk parallel computation. But um, all kinds of issues. Also stare at Concurrency for LLVM. If they can salvage async, would be amazing. Cancel blockage, function pointers, measuring memory, hidden control flow, coloring. Ya, I wanna see. There's no roll your own to reduce syscalls. It's a nightmare. He would be the one to fix it though.
Per user DB... I do not feel that at all. Have fun managing that in some multi instance cluster environment. A normal database is 10x easier
Per user db is bullshit in 99% cases, but there might be 1% cases where it makes sense. Like some online photoshop, or other offline-like tools used in online.
I don't get not reinventing the wheel argument, because we don't have cars with wooden wheels. The risk to reward ratio has to be worthwhile.
we need kernel developers to use tiger beetle style. Just caught up on the rust/kernel drama from a few months ago and it is sad
"Microseconds did not matter..." My Z80 would like to have a chat with you.
"That's why I'm personally inventing kubernetes myself right now"
Where could we see a stream where the CEO of Turso, the CEO of TigerBeetle and Primagen team-up in the limp-biscuit-Olympics?
Why is SQLite in title when whle stori is about something else? SQLite is just side mentioned.
Squeal lite?
KimStyle method implies keyboard, mouse and handgun
I haven't used multi tenancy in production yet because too much overhead maybe I haven't implemented the correct pattern but I would appreciate really super simple multi tenancy I think I came across the same issues as these dudes
Can someone point me to the name of the series(?) where he was writing the simulation testing code? (not the original tiger beetle interview)
Greetings from the Netherlands, the Primeagen. I am not sure if you noticed the acrostic in the section "Appendix: Why not Zig?" Look at the boldly printed first letter of each sentence.
yes i noticed it right away
@@ThePrimeTimeagen Excellent, I should have known
Didn't understand anything.. what the hell is a Turso. I thought the video was about Sqlite :(
22:31 - Time sharing just exist. Rust "programmers" (ex-Java Script) exploring the world.
What the point to add backdoor (Rust) to good DB?
OT: If the Primagen would be reacting to the new biggest found prime number, how would he sign off that video? "The name is the Mersenne Primagen!"
Is ssh coffee shop a "frontendless frontend"?
7:13 "ready or not, here i come, socket write" :D
_sorry, had too_
The SQLite project does not accept patches from people who have not submitted an affidavit dedicating their contribution into the public domain. This means that Turso does not want to submit an affidavit dedicating their contributions into the public domain.
isn't it easier to contribute to their fork because you can just open a PR?
Or it means that Turso might want to implement features that are specific to their use case and are unlikely to be accepted for merge into SQLite. Also, fyi, Turso devs have actually up streamed changes back to SQLite.
The requirements to submit to sqlite are WAY deeper than just a "this is public domain code"
Shorthand? I thought you didn't mind typing!
The Shillagean with this weeks Shillogism
No cold start can be handled the same way android handles its VMs. You basically pre-fork the initialized base VM (zygot in Android speak IIRC). This way, you're always connecting to a pre-warmed image rather than cold starting on each need.
I don't like a lot of things about k8s or docker... or ansible/tower etc. reinventing all the stuff as well... including server, security, auth, rpc, protocols etc etc etc. Why. Been trying to bring sim testing into an app layer 4evar... but it's never the app that's the issue...
Ansible and Salt are great tools to ensure identical system config if one doesn't use Nix.
Those and Terraform/OpenTofu means you truly can rebuild your whole infrastructure quickly when disaster happens.
@PanduPoluan yeah... but with some serious limitations and vulnerabilities... it's why I'm bothering to rewrite em...smh.
DST in pt-br is Doença Sexualmente Transmissível (english’s STD)
Great acronym, love it
I think Elixir might be the only language that does async right. And other BEAM languages, of course.
I have seen such kind deadlicks because of OS changing thread. To me was solution to go shorter timeslots so that probability goes down
The solution was that time 20 yesrs ago use someyhing faster than os thread switch that time that was directx timers. so we had like 240 voip streams + control sent and recieved 30ms cunks while os alowed only 1ms delta. Mutex in hardware interupt callback caused trouble.
Sounds like TLA+, without TLA+