Hi fiend, Techie. Thank you for sharing knowledges. Watched this video I don't see you mention to how to Spring cloud dataflow binding with a Message Broker (in here you are using Apache Kafka). If possible please explain this point.
Hi Basant, If you have any idea on CDC (Change Data Capture) , can you create a simple video for data migration in real time by using spring boot, kafka, kafka connect and debezium with a simple my-sql data base. There is no proper example given in GOOGLE with this approach. Hence i recollected your name and thought you are the right person to make this video. I have full confidence on you that you can make it. God bless you Basant. I know this is not easy.. This is the important question coming in all interviews ..
Q. Here we are not mentioning any topic name in any of the properties file how it is able to communicate without topic name and if we want to mention perticular topic name how can we do that? can you please answer this ❤? Thank you sir. for viseos like this ❤❤
Nice stuff! I followed your instruction, once registered app using "add register" command, i can see the app in dashboard. However, it failed with ArtifcatNotFound exception. So i have couple of questions 1. Should we keep app JAR in the same location where cloud-data-flow JAR? 2. How do we form Maven URI for our JARs?
Thanks for this detailed explanation. I did not understand the real purpose of the Cloud Data Flow server. Without this server, if I start the 3 microservices individually then also the product stream created by the 1st would be consumed by the 2nd microservice because it is subscribed to that topic. Same way, the discount stream from the 2nd would be consumed by the 3rd microservice. So, what is the real use of this Cloud Data Server. Could you please elaborate?
Hi, I would like to say thanks for your video, this is very clear & helpful to me. Is this flow olap supported, and fast enough to handle million records? Please advise.
so, if I want to change the Book dto to add a new field , I need to change in all three services. isnt this kind of coupling I should avoid ? great presentation. thanks
Its an awesome tutorial, very nice explanation. how do we implement microservice data orchestration based / choreography based saga design pattern. can you please throw some light will be highly appreciated. Thanks in advance
Thanks codingAndCaffeine , I really happy to do that but it's not possible to complete in one single video , people will get bore even same for me . But as I mentioned when I will start building full microservice series that time I will try to cover this scenario. Once again thank you so much for regularly following javatechie 👏
Hi It is great and working nice.. But how can we configure different Topic names I tried in Source it is throwing error.. Do you know how to handle this ?
@@Javatechie 2020-12-10 15:26:17.554 WARN 17608 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'processingCl.addProducts.inboundChannelAdapter'; nested exception is org.springframework.messaging.core.DestinationResolutionException: failed to look up MessageChannel with name 'myowntopic' in the BeanFactory.; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'myowntopic' available 2020-12-10 15:26:17.555 INFO 17608 --- [ main] o.s.s.c.ThreadPoolTaskScheduler : Shutting down ExecutorService 'taskScheduler' 2020-12-10 15:26:17.559 INFO 17608 --- [ main] ConditionEvaluationReportLoggingListener : Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. 2020-12-10 15:26:17.608 ERROR 17608 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter : *************************** APPLICATION FAILED TO START *************************** Description: A component required a bean named 'myowntopic' that could not be found. Action: Consider defining a bean named 'myowntopic' in your configuration.
java.base@10.0.1/java.net.ServerSocket.implAccept(ServerSocket.java:551) java.base@10.0.1/java.net.ServerSocket.accept(ServerSocket.java:519) org.h2.server.TcpServer.listen(TcpServer.java:247) org.h2.tools.Server.run(Server.java:576) java.base@10.0.1/java.lang.Thread.run(Thread.java:844) 2020-03-24 18:43:55.879 ERROR 13993 --- [ main] o.s.boot.SpringApplication : Application startup failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1631) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1080) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:857) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) at org.springframework.cloud.dataflow.server.local.LocalDataFlowServer.main(LocalDataFlowServer.java:36) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException at org.hibernate.boot.spi.XmlMappingBinderAccess.(XmlMappingBinderAccess.java:43) at org.hibernate.boot.MetadataSources.(MetadataSources.java:87) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.(EntityManagerFactoryBuilderImpl.java:208) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.(EntityManagerFactoryBuilderImpl.java:163) at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:54) at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:360) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:384) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:371) at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:336) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1689) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1627) ... 24 common frames omitted Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:466)
Nice but does this work with latest Spring Boot 3.1.x? EnableBinding is not supported in 3.x. JDK is 1.8. 4) Download Spring Cloud Data Flow Server jar Download. This file is not there on the path. Maven jar on JDK 17 gives Manifest attribute issue java -jar spring-cloud-dataflow-shell-1.3.0.M1.jar 2023-07-27T17:24:32+0530 ERROR main o.s.b.SpringApplication:771 - Application startup failed java.lang.IllegalStateException: Cannot load configuration class: org.springframework.cloud.dataflow.shell.app.ShellApplication at org.springframework.context.annotation.ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:403) at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanFactory(ConfigurationClassPostProcessor.java:249) at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:281) at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:125) at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
Hi fiend, Techie. Thank you for sharing knowledges. Watched this video I don't see you mention to how to Spring cloud dataflow binding with a Message Broker (in here you are using Apache Kafka). If possible please explain this point.
Very good one. Very crisp. The restart of the data flow server should not have been done
Very nice tutorial
Thanks
Hi Basant, If you have any idea on CDC (Change Data Capture) , can you create a simple video for data migration in real time by using spring boot, kafka, kafka connect and debezium with a simple my-sql data base. There is no proper example given in GOOGLE with this approach. Hence i recollected your name and thought you are the right person to make this video. I have full confidence on you that you can make it. God bless you Basant. I know this is not easy.. This is the important question coming in all interviews ..
Thanks Raghu , definitely I will check this out and try to find out the content with proper understanding
Q. Here we are not mentioning any topic name in any of the properties file how it is able to communicate without topic name and if we want to mention perticular topic name how can we do that? can you please answer this ❤?
Thank you sir.
for viseos like this ❤❤
Nice stuff!
I followed your instruction, once registered app using "add register" command, i can see the app in dashboard. However, it failed with ArtifcatNotFound exception. So i have couple of questions
1. Should we keep app JAR in the same location where cloud-data-flow JAR?
2. How do we form Maven URI for our JARs?
In case of REST API how response is received after processing all the individual MS. Since data flow is an one way communication.
in this example where you have specified KAFKA server details ?
Not required if it's running on default
thanks a lot for this tuto 👍🏻
Beautifully done man!!! Thanks
Thanks for this detailed explanation. I did not understand the real purpose of the Cloud Data Flow server.
Without this server, if I start the 3 microservices individually then also the product stream created by the 1st would be consumed by the 2nd microservice because it is subscribed to that topic.
Same way, the discount stream from the 2nd would be consumed by the 3rd microservice.
So, what is the real use of this Cloud Data Server. Could you please elaborate?
nice.
Hi, I would like to say thanks for your video, this is very clear & helpful to me.
Is this flow olap supported, and fast enough to handle million records? Please advise.
To process huge number of data we need to implement completableFuture or some async mechanism
so, if I want to change the Book dto to add a new field , I need to change in all three services. isnt this kind of coupling I should avoid ? great presentation. thanks
Yes partially correct but to avoid that you can use schema registry
@@Javatechie thanks JT. I didn't know about this. Really, really appreciate all your content. Keep going !
@@grrlgd3835 no problem already i uploaded this scheme registry avro concept please check
@@Javatechie will do thanks
Thank u so much , please upload more video with real time project
I realize it is kind of off topic but do anyone know of a good website to stream newly released movies online?
Its an awesome tutorial, very nice explanation. how do we implement microservice data orchestration based / choreography based saga design pattern. can you please throw some light will be highly appreciated. Thanks in advance
excellent, more video please, create a social developer network project with the stack, Spring, JPA, postgres, docker etc thanks a lot
Thanks codingAndCaffeine ,
I really happy to do that but it's not possible to complete in one single video , people will get bore even same for me .
But as I mentioned when I will start building full microservice series that time I will try to cover this scenario.
Once again thank you so much for regularly following javatechie 👏
Excellent
good stuff!
use lombok for the logger :)
Good Work. Can we have multiple Transformers?
Yes we can
nice one !!!! thanks a tonne for this .. !!!
Thanks sir. So this is for one-way communication right???
Yes
@@Javatechie is it impossible to use two way communication with cloud data flow?
Thanks Sir, Can it be like source to source and then processor. Or source to processor and one more processor?
Yes we can add multiple component
Ok thanks, can you please make videos on task deployment in spring cloud data flow and how to trigger it?
thank u sooo much sir
Hi
It is great and working nice..
But how can we configure different Topic names
I tried in Source it is throwing error..
Do you know how to handle this ?
That we need to configure in yml file .
@@Javatechie even I tried directly like
@InputChannelAdapter(value="myowntopic")
Even for this it is asking bean not found Exception
Can you share complete exception stack trace
@@Javatechie 2020-12-10 15:26:17.554 WARN 17608 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'processingCl.addProducts.inboundChannelAdapter'; nested exception is org.springframework.messaging.core.DestinationResolutionException: failed to look up MessageChannel with name 'myowntopic' in the BeanFactory.; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'myowntopic' available
2020-12-10 15:26:17.555 INFO 17608 --- [ main] o.s.s.c.ThreadPoolTaskScheduler : Shutting down ExecutorService 'taskScheduler'
2020-12-10 15:26:17.559 INFO 17608 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-12-10 15:26:17.608 ERROR 17608 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
A component required a bean named 'myowntopic' that could not be found.
Action:
Consider defining a bean named 'myowntopic' in your configuration.
Can you list down all the topics which you created in Kafka console.
Just verify it creates or not
I am getting am error when I start spring cloud server
java.base@10.0.1/java.net.ServerSocket.implAccept(ServerSocket.java:551)
java.base@10.0.1/java.net.ServerSocket.accept(ServerSocket.java:519)
org.h2.server.TcpServer.listen(TcpServer.java:247)
org.h2.tools.Server.run(Server.java:576)
java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
2020-03-24 18:43:55.879 ERROR 13993 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1631)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1080)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:857)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107)
at org.springframework.cloud.dataflow.server.local.LocalDataFlowServer.main(LocalDataFlowServer.java:36)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
at org.hibernate.boot.spi.XmlMappingBinderAccess.(XmlMappingBinderAccess.java:43)
at org.hibernate.boot.MetadataSources.(MetadataSources.java:87)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.(EntityManagerFactoryBuilderImpl.java:208)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.(EntityManagerFactoryBuilderImpl.java:163)
at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:54)
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:360)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:384)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:371)
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:336)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1689)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1627)
... 24 common frames omitted
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:466)
how to add maven uri:maven//{}
Didn't get your question
Nice but does this work with latest Spring Boot 3.1.x? EnableBinding is not supported in 3.x. JDK is 1.8. 4) Download Spring Cloud Data Flow Server jar Download. This file is not there on the path. Maven jar on JDK 17 gives Manifest attribute issue
java -jar spring-cloud-dataflow-shell-1.3.0.M1.jar
2023-07-27T17:24:32+0530 ERROR main o.s.b.SpringApplication:771 - Application startup failed
java.lang.IllegalStateException: Cannot load configuration class: org.springframework.cloud.dataflow.shell.app.ShellApplication
at org.springframework.context.annotation.ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:403)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanFactory(ConfigurationClassPostProcessor.java:249)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:281)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:125)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
Kindly remind me tomorrow over email (javatechie4u@gmail.com) have already done this using spring boot 3 . I will share the GitHub link of it
@@Javatechie I am also facing the same problem.. could you pls send the github link of Spring boot 3 version.