Thank you so much for taking the time to develop such videos. As a request could you upload the CCA course videos more often as many people plan to take up the CCA exam in january.
Hello Durga sir, I wanted to try the sqoop tutorial in this video. First I just wanted to ls the hdfs directories. so I issued the below command. [cloudera@quickstart ~]$ hadoop fs -ls / ls: Failed on local exception: java.net.SocketException: Network is unreachable; Host Details : local host is: "quickstart.cloudera/10.0.2.15"; destination host is: "quickstart.cloudera":8020; though ps -fu hdfs says both namenode and data node is running. I checked the status using the service command. [cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode status Hadoop namenode is not running [FAILED]
Thinking all the problems will be resolved if I restart all the services, I executed the below command. [cloudera@quickstart conf]$ sudo /home/cloudera/cloudera-manager --express --force [QuickStart] Shutting down CDH services via init scripts... [QuickStart] Disabling CDH services on boot... [QuickStart] Starting Cloudera Manager daemons... [QuickStart] Waiting for Cloudera Manager API... [QuickStart] Configuring deployment... Submitted jobs: 92 [QuickStart] Deploying client configuration... Submitted jobs: 93 [QuickStart] Starting Cloudera Management Service... Submitted jobs: 101 [QuickStart] Enabling Cloudera Manager daemons on boot... Now I thought all services will be up so again checked the status of namenode service. Again it came failed. [cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode status Hadoop namenode is not running [FAILED] Now I decided to manually stop and start the namenode service. Again not much use. [cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode stop no namenode to stop Stopped Hadoop namenode: [ OK ] [cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode status Hadoop namenode is not running [FAILED] [cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode start starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-quickstart.cloudera.out Failed to start Hadoop namenode. Return value: 1 [FAILED] I checked the file /var/log/hadoop-hdfs/hadoop-hdfs-namenode-quickstart.cloudera.out . It just said below log4j:ERROR Could not find value for key log4j.appender.RFA log4j:ERROR Could not instantiate appender named "RFA". I also checked /var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-quickstart.cloudera.log.out . Found below when I searched for error. Can you please suggest me what is the best way to get the services back on track. Unfortunately I am not able to access cloudera manager from browser. Anything that I can do from command line? It would be great if you can help me because of HDFS not able to access I am not able to do much in VM. Probably I am trying too hard, there might be an easy way out. please suggest.
2016-02-24 21:02:48,105 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[NAMENODE], CATEGORY=[LOG_MESSAGE], ROLE=[hdfs-NAMENODE], SEVERITY=[IMPORTANT], SERVICE=[hdfs], HOST_IDS=[quickstart.cloudera], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOSTS=[quickstart.cloudera], EVENTCODE=[EV_LOG_EVENT]}, content=Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories!, timestamp=1456295437905} - 1 of 17 failure(s) in last 79302s java.io.IOException: Error connecting to quickstart.cloudera/10.0.2.15:7184 at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249) at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:198) at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:133) at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.checkSpecificRequestor(AvroEventStorePublishProxy.java:122) at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.publishEvent(AvroEventStorePublishProxy.java:196) at com.cloudera.cmf.event.publish.EventStorePublisherWithRetry$PublishEventTask.run(EventStorePublisherWithRetry.java:242) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketException: Network is unreachable
+itversity sure I will give one more try. I did run cloud era manager from terminal, I ran it as express. unfortunately it did not bring up the service. I thought of trying dfs-start.sh and stop scripts. would it be available as part of cloud Era vm ? I could not find them. As of now I have 8 gb ram laptop and most of the times cloudera manager portal won't come up.
+pavan nama I am also not sure. What is the memory configuration for VM? Are you using --force while bringing up the VM, if it is less than 8 GB for the VM?
+pavan nama Hello Durga sir, Not sure what was the problem. I did shut down the vm and started again next day. Services seem to be up I am able to use the VM. Probably if nothing is working out, better solution is to shutdown I guess :)
when creating directory getting error cannot create directory its in safemode.sudo -u hdfs hdfs dfsadmin -safemode leave used this command but still even if the safe mode is turned off.not able to make directory like this ( hadoop fs -mkdir /user/cloudera/scoop_import. can you please help me with this issue
For any technical discussions or doubts, please use our forum - discuss.itversity.com For practicing on state of the art big data cluster, please sign up on - labs.itversity.com Lab is under free preview until 12/31/2016 and after that subscription charges are 14.99$ per 31 days, 34.99$ per 93 days and 54.99$ per 185 days
Hi Durga, I am not finding name node,data node,etc when i m trying to use "ps -fu hdfs/ps -fu yarn" .And when i m working sqoop commands , facing an error "connection refused".
To test that, the connection is established and able to see/find DB, Tables and Data before integrating pipeline setup between Sources to Hive/HDFS. And also it facilitates Query Column Selectivity based data transfer from Source DB to Target.
Hi Sir I got 6 months plan of big data labs. Can I work on this video using the big data labs. I tried your commands in gw01 hosts but it wasn’t worked. Can you please help me
Hi Durga garu, cloudera 5.4 I get confused request error. already one your video you had explained it.But I tried from command line sudo service hadoop-hdfs-namenode start . command start and end.But for loopback ifconfig 127.0.0.1 is callback throwing error.please help how keep eth1 ifconfig as default. All service are failing because of this.namenode is not starting.pls help.
I have tried listing databases and tables using the "jdbc:mysql://127.0.0.1/retail_db" which worked fine however when am trying with "jdbc:mysql://quickstart.cloudera:3360" its not working for me can you please explain the difference between the 2 commands and how to get exact address for 2 nd command. Thanks a lot for your valuable help.
+Vegesna Naveen Prasad Raju Your port number is incorrect while using quickstart.cloudera. It should be jdbc:mysql://quickstart.cloudera:3306. MySql by default runs on 3306. It is fine to discard port number like you have done using localhost/127.0.0.1, but you cannot type incorrect port number. In this case both quickstart.cloudera or 127.0.0.1 or localhost or ip (ifconfig -a) are same as long as you have correct port number.
i got that i am connecting to mysql but some error is thrown saying that your host PC is not allowing to connect mysql server.while executing sqoop import cmd
Hello Sir, I am trying to eval the statement but i am getting an error cloudera@quickstart ~]$ sqoop eval --connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" --username retail_dba --password cloudera --query"select * from departments" Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 16/02/20 12:53:57 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0 16/02/20 12:53:57 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/02/20 12:53:57 ERROR tool.BaseSqoopTool: Error parsing arguments for eval: 16/02/20 12:53:57 ERROR tool.BaseSqoopTool: Unrecognized argument: --queryselect * from departments This is the link for screenshot gyazo.com/b9c6bdf672e5fbd3665fd52a40aae475 Please help me
+Aditya Gaur Error is clear. You do not have space between --query and "select * from departments". This is the error reported "Unrecognized argument: --queryselect * from departments"
While I am practicing I used this "jdbc:mysql://quickstart.cloudera:3306" and I also used this "jdbc:mysql://quickstart.cloudera" both gave me the same result. What is the exact use of :3306
It is the standard port number for mysql db, this means mysql is using the standard port, so it is optional unless the port is explicitly changed by DBA, it will fail.
close enough, but you always use this 3306 port unless something else explicitly given by admins, so it is a good idea to include this port when you're connecting to mysql -- don't hung up on these nuances, you get enlightened on all these specifics once you keep going. Good catch, BTW.
Hi Durga Sir, Could you please also tell us, how to get and locate the ip:port conncet of "jdbc:mysql://quickstart.cloudera:3360" to connect with Sqoop
Hello Durga Sir, I am planning to give cca175 next month.. but the syllabus looks like changed. Practice + material provided by you is enough? I see the spark version in videos is older one .. does it affect the exam? Is there any site where we can get latest dumps or is there any book available which I can refer ? Please guide me on this. Thanks in advance..
Hi Sir, I'm facing this issue, dont know how too resolve it... Kindly help me While Im trying to evaluate the sqoop command to check and list the databases, its throwing the below error "class path contains multiple slf4j bindings" What should i do now sir...? please help
Hello Durga Sir, I have installed cloudera-quickstart-vm-5.5.0-0-vmware. After logged in i tried to connect mysql using retail_db and cloudera password, i am getting access denied error. Please help. Also I tried jps but no hadoop daemons are running. i tried start-all.sh but it didn't work. Any direction would be great.
+Murali Rachakonda jps command only shows java processes running by that user. In Cloudera VM, HDFS processes will be run by OS user hdfs. Correct command to use - ps -fu or ps -ef|grep -i datanode or any other daemon process. Please send me the screenshot about access issue.
for the last 3 months i'm searching for the notes . thanks a Lot for the proper notes in hithub . thanks a lot durga
Thank you so much for taking the time to develop such videos.
As a request could you upload the CCA course videos more often as many people plan to take up the CCA exam in january.
Excellent Tutorial. Please keep up the good work !
Very detailed explanation, thanks for taking time for preparing videos.
Hello Durga sir,
I wanted to try the sqoop tutorial in this video. First I just wanted to ls the hdfs directories. so I issued the below command.
[cloudera@quickstart ~]$ hadoop fs -ls /
ls: Failed on local exception: java.net.SocketException: Network is unreachable; Host Details : local host is: "quickstart.cloudera/10.0.2.15"; destination host is: "quickstart.cloudera":8020;
though ps -fu hdfs says both namenode and data node is running. I checked the status using the service command.
[cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode status
Hadoop namenode is not running [FAILED]
Thinking all the problems will be resolved if I restart all the services, I executed the below command.
[cloudera@quickstart conf]$ sudo /home/cloudera/cloudera-manager --express --force
[QuickStart] Shutting down CDH services via init scripts...
[QuickStart] Disabling CDH services on boot...
[QuickStart] Starting Cloudera Manager daemons...
[QuickStart] Waiting for Cloudera Manager API...
[QuickStart] Configuring deployment...
Submitted jobs: 92
[QuickStart] Deploying client configuration...
Submitted jobs: 93
[QuickStart] Starting Cloudera Management Service...
Submitted jobs: 101
[QuickStart] Enabling Cloudera Manager daemons on boot...
Now I thought all services will be up so again checked the status of namenode service. Again it came failed.
[cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode status
Hadoop namenode is not running [FAILED]
Now I decided to manually stop and start the namenode service. Again not much use.
[cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode stop
no namenode to stop
Stopped Hadoop namenode: [ OK ]
[cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode status
Hadoop namenode is not running [FAILED]
[cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode start
starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-quickstart.cloudera.out
Failed to start Hadoop namenode. Return value: 1 [FAILED]
I checked the file /var/log/hadoop-hdfs/hadoop-hdfs-namenode-quickstart.cloudera.out . It just said below
log4j:ERROR Could not find value for key log4j.appender.RFA
log4j:ERROR Could not instantiate appender named "RFA".
I also checked /var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-quickstart.cloudera.log.out . Found below when I searched for error. Can you please suggest me what is the best way to get the services back on track. Unfortunately I am not able to access cloudera manager from browser. Anything that I can do from command line? It would be great if you can help me because of HDFS not able to access I am not able to do much in VM. Probably I am trying too hard, there might be an easy way out. please suggest.
2016-02-24 21:02:48,105 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[NAMENODE], CATEGORY=[LOG_MESSAGE], ROLE=[hdfs-NAMENODE], SEVERITY=[IMPORTANT], SERVICE=[hdfs], HOST_IDS=[quickstart.cloudera], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOSTS=[quickstart.cloudera], EVENTCODE=[EV_LOG_EVENT]}, content=Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories!, timestamp=1456295437905} - 1 of 17 failure(s) in last 79302s
java.io.IOException: Error connecting to quickstart.cloudera/10.0.2.15:7184
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:198)
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:133)
at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.checkSpecificRequestor(AvroEventStorePublishProxy.java:122)
at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.publishEvent(AvroEventStorePublishProxy.java:196)
at com.cloudera.cmf.event.publish.EventStorePublisherWithRetry$PublishEventTask.run(EventStorePublisherWithRetry.java:242)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Network is unreachable
+pavan nama Try restarting the services from Cloudera Manager. I have shown as part of common issues towards the end of the playlist.
+itversity sure I will give one more try. I did run cloud era manager from terminal, I ran it as express. unfortunately it did not bring up the service. I thought of trying dfs-start.sh and stop scripts. would it be available as part of cloud Era vm ? I could not find them. As of now I have 8 gb ram laptop and most of the times cloudera manager portal won't come up.
+pavan nama I am also not sure. What is the memory configuration for VM? Are you using --force while bringing up the VM, if it is less than 8 GB for the VM?
+itversity yes I did use -force. I have assigned 5.5gb ram and 2 cores to the vm. I will again try today evening if I can bring up the services.
+pavan nama Hello Durga sir, Not sure what was the problem. I did shut down the vm and started again next day. Services seem to be up I am able to use the VM. Probably if nothing is working out, better solution is to shutdown I guess :)
when creating directory getting error cannot create directory its in safemode.sudo -u hdfs hdfs dfsadmin -safemode leave used this command but still even if the safe mode is turned off.not able to make directory like this ( hadoop fs -mkdir /user/cloudera/scoop_import. can you please help me with this issue
For any technical discussions or doubts, please use our forum - discuss.itversity.com
For practicing on state of the art big data cluster, please sign up on - labs.itversity.com
Lab is under free preview until 12/31/2016 and after that subscription
charges are 14.99$ per 31 days, 34.99$ per 93 days and 54.99$ per 185 days
I tried exactly the same procedure...but I am getting "No sqoop tool:list. See 'sqoop help'". Kindly help
How to contact to you sir . i want to learn bigdata practical course . pls help me sir
Hi Durga, I am not finding name node,data node,etc when i m trying to use "ps -fu hdfs/ps -fu yarn" .And when i m working sqoop commands , facing an error "connection refused".
+Saikrishna Manda Please watch this video: th-cam.com/video/GJ-k5ntKMLg/w-d-xo.html
this videos are useful for hortonworks hdpcd certification ?
Hi Sir, I don't see any use of eval in production. Could you please tell me why would someone use sqoop to query DB? People can directly access DB.
To test that, the connection is established and able to see/find DB, Tables and Data before integrating pipeline setup between Sources to Hive/HDFS. And also it facilitates Query Column Selectivity based data transfer from Source DB to Target.
Hi Sir I got 6 months plan of big data labs. Can I work on this video using the big data labs. I tried your commands in gw01 hosts but it wasn’t worked. Can you please help me
Hi Durga garu,
cloudera 5.4 I get confused request error.
already one your video you had explained it.But I tried from command line
sudo service hadoop-hdfs-namenode start .
command start and end.But for loopback ifconfig 127.0.0.1 is callback throwing error.please help how keep eth1 ifconfig as default.
All service are failing because of this.namenode is not starting.pls help.
I have tried listing databases and tables using the "jdbc:mysql://127.0.0.1/retail_db" which worked fine however when am trying with "jdbc:mysql://quickstart.cloudera:3360" its not working for me
can you please explain the difference between the 2 commands and how to get exact address for 2 nd command.
Thanks a lot for your valuable help.
+Vegesna Naveen Prasad Raju Your port number is incorrect while using quickstart.cloudera. It should be jdbc:mysql://quickstart.cloudera:3306. MySql by default runs on 3306. It is fine to discard port number like you have done using localhost/127.0.0.1, but you cannot type incorrect port number. In this case both quickstart.cloudera or 127.0.0.1 or localhost or ip (ifconfig -a) are same as long as you have correct port number.
***** thank you Sir
Hi, I just noticed that the linkedin hadoop certifications group is renamed to 'IT Versity - Certifications'
+Shaik Muhammad Yes, that is right.
Your videos are great. Thank you so much.
hello sir i use biginsight while practicing sqoop import command what port no i have to write for each different command.
Which database are you trying to connect?
i got that i am connecting to mysql but some error is thrown saying that your host PC is not allowing to connect mysql server.while executing sqoop import cmd
Hello Sir,
I am trying to eval the statement but i am getting an error
cloudera@quickstart ~]$ sqoop eval --connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" --username retail_dba --password cloudera --query"select * from departments"
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/02/20 12:53:57 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0
16/02/20 12:53:57 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/02/20 12:53:57 ERROR tool.BaseSqoopTool: Error parsing arguments for eval:
16/02/20 12:53:57 ERROR tool.BaseSqoopTool: Unrecognized argument: --queryselect * from departments
This is the link for screenshot
gyazo.com/b9c6bdf672e5fbd3665fd52a40aae475
Please help me
+Aditya Gaur Error is clear. You do not have space between --query and "select * from departments". This is the error reported "Unrecognized argument: --queryselect * from departments"
+itversity AHH Silly Mistake.Thanks for the quick response
While I am practicing I used this "jdbc:mysql://quickstart.cloudera:3306" and I also used this "jdbc:mysql://quickstart.cloudera" both gave me the same result. What is the exact use of :3306
It is the standard port number for mysql db, this means mysql is using the standard port, so it is optional unless the port is explicitly changed by DBA, it will fail.
Thanks. So in the real environment the port number will change and we need to get that info from admin. Am I right???
close enough, but you always use this 3306 port unless something else explicitly given by admins, so it is a good idea to include this port when you're connecting to mysql -- don't hung up on these nuances, you get enlightened on all these specifics once you keep going. Good catch, BTW.
thanks
superb..keep on posting
Excellent video
Hi Durga Sir,
Could you please also tell us, how to get and locate the ip:port conncet of "jdbc:mysql://quickstart.cloudera:3360" to connect with Sqoop
URL should be "jdbc:mysql://quickstart.cloudera:3306" - 3306 is default port number for mysql
Hello Durga Sir,
I am planning to give cca175 next month.. but the syllabus looks like changed. Practice + material provided by you is enough?
I see the spark version in videos is older one .. does it affect the exam?
Is there any site where we can get latest dumps or is there any book available which I can refer ?
Please guide me on this. Thanks in advance..
You can prepare by going through this playlist - th-cam.com/play/PLf0swTFhTI8q0x0V1E6We5zBQ9UazHFY0.html
So I have to follow both the playlist to clear the exam ??
No, just the latest one.
Hi, is this updated playlist enough for CCA175 certification?
Hi Sir,
I'm facing this issue, dont know how too resolve it... Kindly help me
While Im trying to evaluate the sqoop command to check and list the databases, its throwing the below error
"class path contains multiple slf4j bindings"
What should i do now sir...? please help
Hi Sir, I've used mysql connector-java 5.1.36.jar, dont know whats wrong I've done.
Can you send the complete error log to my email id training@itversity.com?
Hi Durga sir,
I've the error screen shot for your kind perusal...
With kind regards,
Varadharaj.S
I've sent the error screen shot for your kind perusal...
Hello Sir,
Unable to create sqoop_import directory. Error says: name node is in safe mode
Kindly help me
use sudo hadoop fs -mkdir /users/cloudera/sqoop_import
goto cloudera manager console and restart sqoop
This is fantastic. Thanks a lot!
Hello Durga Sir, I have installed cloudera-quickstart-vm-5.5.0-0-vmware. After logged in i tried to connect mysql using retail_db and cloudera password, i am getting access denied error. Please help. Also I tried jps but no hadoop daemons are running. i tried start-all.sh but it didn't work. Any direction would be great.
+Murali Rachakonda jps command only shows java processes running by that user. In Cloudera VM, HDFS processes will be run by OS user hdfs. Correct command to use - ps -fu or ps -ef|grep -i datanode or any other daemon process. Please send me the screenshot about access issue.
+itversity I am very sorry there was typo in the username. I used retail_db instead of retail_dba
Thanks a lot Sir...!! really good explanation, Appreciate your help :)
how to know the username in sqoop.
hello sir..is it possible to share git hub credentials?
i want to run the codes.
+Veeru I.S My github is public. You should be able to access it.
Ho..is.it..thank you sir.
+Veeru I.S Are you able to access it?
yes..sir..thank you.
Awesome. Thank you