Thanks Steven. Excellent video. The RetryFlowFile is definitely a good option specially in the case of server\connection is down. I thought of doing the same thing for other failures with the PutFile processor but faced the same challange of creating unique & informative file name but the bigger challange in case of PuSql Failure is how you are going to spit the complete sql query into a file for future recovery specially when the flowfile is parameterized sql and the different parameters are provided as attributes (sql.args.N....). I found myself having to write an ExecuteScript basically to alter the content of the flowfile and reassign the actual values. Not sure if you have another thought on that. Thanks
Hello Steven, your videos are very good. It would be good to see a video explaining how to do a good nifi tuning to reduce execution times. I work with the tool in cluster format and I have had problems moving quickly and effectively (100 million records), I do all the tuning on the query and in my dbeaver it takes an acceptable ad-hoc time, but in nifi it takes approximately 4 times longer to execute the same query and I require it to implement it in a database of a legacy system to a DWH, since the time to make the replicas should not be greater than 40 minutes.
Dear sir, my ExecuteSQL processor has successfully properties verification, but it still shows invalid and not running. Can you help me with some ideas to debug what went wrong?
How do you make Penalty Duration on the RetryFlowFile processor progressively larger based on the amount of retries? For example first retry is 5 sec. Then it goes 15 s, 30 s, 1 min, 5 min, 15 min, 30 min, 1 h, 3 h and so on...
Check out the processors like "ListenTCP", "ListenUDP" and "ListenWebSocket". There are a couple others and see if they are what your looking for. nifi.apache.org/docs.html
thanks for the video! Coming from Nifi 1.6.0, totally missed such a Processor.. finally!
Glad it was helpful to you.
Thanks Steven. Excellent video. The RetryFlowFile is definitely a good option specially in the case of server\connection is down. I thought of doing the same thing for other failures with the PutFile processor but faced the same challange of creating unique & informative file name but the bigger challange in case of PuSql Failure is how you are going to spit the complete sql query into a file for future recovery specially when the flowfile is parameterized sql and the different parameters are provided as attributes (sql.args.N....). I found myself having to write an ExecuteScript basically to alter the content of the flowfile and reassign the actual values. Not sure if you have another thought on that. Thanks
Hello Steven, your videos are very good. It would be good to see a video explaining how to do a good nifi tuning to reduce execution times. I work with the tool in cluster format and I have had problems moving quickly and effectively (100 million records), I do all the tuning on the query and in my dbeaver it takes an acceptable ad-hoc time, but in nifi it takes approximately 4 times longer to execute the same query and I require it to implement it in a database of a legacy system to a DWH, since the time to make the replicas should not be greater than 40 minutes.
Hi BAMC, Can you please let me know the processors you are using for inserting such huge records? We are using PutSQL and it fails miserably often
Dear sir, my ExecuteSQL processor has successfully properties verification, but it still shows invalid and not running. Can you help me with some ideas to debug what went wrong?
How do you make Penalty Duration on the RetryFlowFile processor progressively larger based on the amount of retries? For example first retry is 5 sec. Then it goes 15 s, 30 s, 1 min, 5 min, 15 min, 30 min, 1 h, 3 h and so on...
thanks for the video but i couldnt add the last processor it needs some permission i guess. Is there any other way to solve this kind of issues ?
Hi, Is there any feasibility to read Modbus TCP/IP data with NiFi?
Check out the processors like "ListenTCP", "ListenUDP" and "ListenWebSocket". There are a couple others and see if they are what your looking for. nifi.apache.org/docs.html
@@StevenKoon Got the solution with PLC4X Processor. Thanks.