+Ankur Varsheny Watch till the end. Your question is answered. The tool he presents is intended for analytics, so live updates are the opposite of what you want. The data needs to be a static snapshot to give comparable results across multiple queries. That's also why warming the caches and taking the shortest runtime is not cheating, because you can expect the analyst to make many consecutive queries that need to access all of the data. Time spent pre-processing the data pays off many times over, because answers is saved later. The whole point here is that they built an optimized tool for their domain, so the generic database has no hope of keeping up, even if it's magic. And to top it all off they did it in Python, which everyone and their mom will tell you is the anti-language for high-throughput, number-crunching applications. Only C, Fortran etc. can be fast like that ;)
blew my mind! very insightful!
Bigtable by google, or just Sorten String can be used for analytic data(GA is using bigtable as a banking storage.
Really helpful,excellent work dude!
To compress the big data matrix, I notice you need to sort it first. How to perform sorting efficiently in this case?
hi if new rows added would it be sorted and processed??
Thanks
Its no more a database, you are querying pre processed data. What will happen when you are to add new rows in your data.
hmm you got a point but i am assuming new rows get processed and sorted as well??
cartman but it will be inefficient at best?
+Ankur Varsheny Watch till the end. Your question is answered. The tool he presents is intended for analytics, so live updates are the opposite of what you want. The data needs to be a static snapshot to give comparable results across multiple queries. That's also why warming the caches and taking the shortest runtime is not cheating, because you can expect the analyst to make many consecutive queries that need to access all of the data. Time spent pre-processing the data pays off many times over, because answers is saved later.
The whole point here is that they built an optimized tool for their domain, so the generic database has no hope of keeping up, even if it's magic. And to top it all off they did it in Python, which everyone and their mom will tell you is the anti-language for high-throughput, number-crunching applications. Only C, Fortran etc. can be fast like that ;)
Json is a shit format.I prefer csv (CSV Com DB is the future)