I struggled for almost 1 hour to find the right content that makes sense about the process of checkPointing and then found this video that explains the concepts very clearly..just a small comment..it would have been even better if you could explained where the FSImage and edit logs are stored whether in in-memory or disk or both?
great explanation....just one query...purpose of check pointing is to reduce the boot up time of NameNode. But when NameNode boots up, fsimage file gets loaded into the memory so does it not take more time to load the fsimage file data into memory because it is also a large file as edit log. and even if check pointing happens, fsimage file will still be large so boot up will still take time ...could you please correct my understanding ?
Checkpointing main purpose to to take snapshot of the file system on regular basis. Also fsimage will be in compact form and already has the complete snapshot of the file system. Time is taken in replaying the transactions from the edit log. Each transaction operation has to be applied on the file system.
what to you mean by "replay"? The operations are already executed by the NameNode & logedit is updated. On bootup, FSImage needs to be updated with delta from EditLog file - Do you mean the same by saying "Replay"?
HI, In case of High Availability, checkpointing is done just to ensure the Authenticity of the fsImage at both Active Name Node and Standby Name Node side. Checkout below video and cloudera blog for details:- th-cam.com/video/5U6EIv_w5Bg/w-d-xo.html blog.cloudera.com/blog/2014/03/a-guide-to-checkpointing-in-hadoop/
0309amit You can configure same machine as Name Node and Checking Pointing Node for demo purpose. For demo purpose I did the same. But, in practice, it is better to configure checkpointing node on different machine...
![Shareปลาวาฬ ส่งท้าย SS1 พาเปียโน ขึ้น Ferrari ก่อนไปขาย แลก Nissan l [Nickynachat]](http://i.ytimg.com/vi/KAknGkTNIvg/mqdefault.jpg)
I have watched many videos on Checkpointing so far and this is the best of all. You are awesome!!
Superb explanation.
this is the most amazing explanation of this concept I have ever come across. Thank you
I struggled for almost 1 hour to find the right content that makes sense about the process of checkPointing and then found this video that explains the concepts very clearly..just a small comment..it would have been even better if you could explained where the FSImage and edit logs are stored whether in in-memory or disk or both?
Crisp and Clear Explanation , Thanks
Very nice
Thanks for your time to make this video so great!
This is the best explanation I have seen, very useful. And I have done Cloudera Admin training! Well done sir!
+mcalnd Thank you.. Spark will soon come..
Checkpoint is explained in detailed and precise way
+Gopi Krishnan Thank You...
WOW, Your information is very complete and neatly said in simplest way. Thanks for your time spent in explaining it :-)
+John Wesley Thank you... Starting Apache Spark very soon...
Really great explanation…
thnx a lot .its help me a lot
Thanks you, It's very good
Neatly explained!!!! thanks!!!! Can you upload full process architecture of YARN.
Thank you. Sorry I don't have Yarn. Now I take Scala and Spark online training....
I have discussed spark on yarn..Probably this will be helpful.
th-cam.com/video/cWt-Y0hQ2RM/w-d-xo.html
Hi, Pramod your teaching method is really easy to understand thank you.I need one help from you please provide me some apache pig videos.
Dont have pig videos. I do now Scala , Spark Kakfa and Cassandra
In Hight availability name node hadoop 2 checkpoint merger old fsimage and all journal node edit logs OR particular journal node it will merge.
great explanation....just one query...purpose of check pointing is to reduce the boot up time of NameNode. But when NameNode boots up, fsimage file gets loaded into the memory so does it not take more time to load the fsimage file data into memory because it is also a large file as edit log. and even if check pointing happens, fsimage file will still be large so boot up will still take time ...could you please correct my understanding ?
Checkpointing main purpose to to take snapshot of the file system on regular basis. Also fsimage will be in compact form and already has the complete snapshot of the file system. Time is taken in replaying the transactions from the edit log. Each transaction operation has to be applied on the file system.
Does the fsimage will become much bigger after several times merged?
Do you have something that talks about Hadoop Federation. Its advantages and disadvantages
Sorry Nitin.. I do have...
don't have
what to you mean by "replay"? The operations are already executed by the NameNode & logedit is updated. On bootup, FSImage needs to be updated with delta from EditLog file - Do you mean the same by saying "Replay"?
Yes..
During checkpoint process where new transaction goes
Hi Pramod, Can you please explain how check point process in hadoop name node high availability. Secondary name node not available.
HI, In case of High Availability, checkpointing is done just to ensure the Authenticity of the fsImage at both Active Name Node and Standby Name Node side.
Checkout below video and cloudera blog for details:-
th-cam.com/video/5U6EIv_w5Bg/w-d-xo.html
blog.cloudera.com/blog/2014/03/a-guide-to-checkpointing-in-hadoop/
fab
+Tanweer Zaki Thank you..
How data comes on secondary name node? is there name node can send or secondary namenode can take data from namenode..
fsimage and edit log is tranferred from NameNode to Secondary NN
If edit logs reached 1millions transaction. How checkpoint node knows
what is check point in case of single Name Node
0309amit You can configure same machine as Name Node and Checking Pointing Node for demo purpose. For demo purpose I did the same. But, in practice, it is better to configure checkpointing node on different machine...
Can you tell me where this video was downloaded from? I want to see what the site has to offer. This video was very helpful.