here in the range end you have put hardcoded value. But if that total count is available in restAPI result, How can I get that to use in the end condition in range
Thank you so much! What if we don't know how many records are in there, or the number of records grows with time, then we can't hard code a End number at 7:24?
When setting up the Range you are hardcoding star=0 and end 1279. How can we use "Dynamic content" for it? I tried using function @range(1, 40) but am getting error "The property 'paginationRules' is invalid or missing.,Source=Microsoft.DataTransfer.ClientLibrary,'"
great tutorial @wafaStudies just wanted to know one thing how to set the end as dynamic? I wanted to use the output of web where I will be picking up __count and wanted to pass it to end range how to do it ?
@@MOHAMMADSHAKIB-d8d Hi Sir, I have a similar problem wherein every call to the API returns a different value We have put in Query parameters variable start, End(empty) count; You are suggesting to put one more condition for EndCondition and leave it empty, How will the Copy activity know the number of times to iterate or when to come out ?.. Request Your valuable inputs please
I did exam same, but rows are restricting to 100 only, I have around 1000+ values, everytime I tried to follow it but I see only 100 rows are written to azure sql, why so?
Hi, what if the offset is not a number but blank in the first api request and returns a long string for each call? Instead of range is it body in the pagination rule?
What if we don't know the end value of range how to handle that scenario. Keeping in mind that we need to create a generic solution which can handle any kind of range say 0-20 or 0-2000 by itself.
When u dont know end value, u can use random high number too i guess. That wsy it will stop automatically. But i will research mkre too if any other way.
Can we set the range value start, end, and offset with dynamic content/variables that have been assigned values? At the beginning, we don't know how many total data there are, so before performing the copy activity, usually I will get the total data first.
@@cusukanya Yes, I found the solution. Fortunately, the API I'm using (SAP CX) supports the parameters count, skip, and top. So, the process I'm following is first counting the data I will process. Next, I set the total paging with the following condition: @string(if(less(int(variables('totaldata')), 2000), 1, if(less(int(variables('totaldata')), 10000), 3, if(less(int(variables('totaldata')), 100000), 10, if(less(int(variables('totaldata')), 200000), 20, 30))))) After obtaining the total paging, the next step is to set the data per paging, until finally, I iterate through each paging using the values skip = totalskipbefore + totaldataperpaging and top = totaldataperpaging
Hi Sir, We have a similar issue at hand ; Just to clarify - You are asking us to add one more pagination rule and it would read as "End Condition" $.count "In the add dynamic content" - should we add something ?. Request to clarify.. thanks in advance
Thanks for the video. How to dynamically add the offset? I have different totalrows for different tables and I want to do the pagination dynamic for all tables
Thank you. Very good and informative. However, I have a problem. My API does not return the total number of records, so I do not know it ahead of time. How to deal with something like this?
here in the range end you have put hardcoded value. But if that total count is available in restAPI result, How can I get that to use in the end condition in range
Thank you so much!
What if we don't know how many records are in there, or the number of records grows with time, then we can't hard code a End number at 7:24?
Did you find solution of problem?
@@Draqqun No I haven't.
Amazing explanation. Pagination was always confusing untill you explained. Thankyou 😊
he has hardcoded the range value hence it is fetching the offset 20. The whole idea was to parameterize the offset value which he could not do it.
When setting up the Range you are hardcoding star=0 and end 1279. How can we use "Dynamic content" for it? I tried using function @range(1, 40) but am getting error "The property 'paginationRules' is invalid or missing.,Source=Microsoft.DataTransfer.ClientLibrary,'"
How do we do this in Data flow? I need to flatten the data in Data flow.
great tutorial @wafaStudies just wanted to know one thing how to set the end as dynamic? I wanted to use the output of web where I will be picking up __count and wanted to pass it to end range how to do it ?
this has been resolved by adding one more pagination rule. - Endcondition = empty and leave the End as blank
@@MOHAMMADSHAKIB-d8d Hi Sir, I have a similar problem wherein every call to the API returns a different value We have put in Query parameters variable start, End(empty) count; You are suggesting to put one more condition for EndCondition and leave it empty, How will the Copy activity know the number of times to iterate or when to come out ?.. Request Your valuable inputs please
I did exam same, but rows are restricting to 100 only, I have around 1000+ values, everytime I tried to follow it but I see only 100 rows are written to azure sql, why so?
Hi, what if the offset is not a number but blank in the first api request and returns a long string for each call? Instead of range is it body in the pagination rule?
What if we don't know the end value of range how to handle that scenario. Keeping in mind that we need to create a generic solution which can handle any kind of range say 0-20 or 0-2000 by itself.
When u dont know end value, u can use random high number too i guess. That wsy it will stop automatically. But i will research mkre too if any other way.
Hi Nitin/Wafastudies , If I have total row count in API result, how can I get that to use in pagination range end condition.Pls help
@wafastudies how do we do this in data that we flatten in data flow? The pagination rules are different in data flow compared to copy activity
@wafastudies my friend where are you
@wafastudies
Did you get solution for this, I need to do the same@@raycharlesvillaruel4383
Can't we pass that 1279 end value as $.count
In my case that number is dynamic I have to take that from json file
did u found anything about this ?
Can we set the range value start, end, and offset with dynamic content/variables that have been assigned values? At the beginning, we don't know how many total data there are, so before performing the copy activity, usually I will get the total data first.
are you able to get a solution for passing it dynamically. am really stuck at passing these values dynamically.
@@cusukanya Yes, I found the solution. Fortunately, the API I'm using (SAP CX) supports the parameters count, skip, and top. So, the process I'm following is first counting the data I will process. Next, I set the total paging with the following condition:
@string(if(less(int(variables('totaldata')), 2000), 1, if(less(int(variables('totaldata')), 10000), 3, if(less(int(variables('totaldata')), 100000), 10, if(less(int(variables('totaldata')), 200000), 20, 30)))))
After obtaining the total paging, the next step is to set the data per paging, until finally, I iterate through each paging using the values skip = totalskipbefore + totaldataperpaging and top = totaldataperpaging
Thank you for the explanation. You can set End with $.count to be dynamic!
Hi Sir, We have a similar issue at hand ; Just to clarify - You are asking us to add one more pagination rule and it would read as "End Condition" $.count "In the add dynamic content" - should we add something ?. Request to clarify.. thanks in advance
Thanks for the video. How to dynamically add the offset? I have different totalrows for different tables and I want to do the pagination dynamic for all tables
what happens if you dont know the size of your data , which often is the case
Thank you. Very good and informative. However, I have a problem. My API does not return the total number of records, so I do not know it ahead of time. How to deal with something like this?
Is this playlist is done?
It may grow based on new features getting released in adf
Thank you so much maheer!! :)
Welcome 😊
Are you going to start GCP videos also bro?
Great bhai
Thank you :-)