Awesome! I was struggling to understand how different measurements are organised after ingested to Timestreams; but thanks to you I have understood it. I actually thought I was doing something wrong and I was having this different rows for each measurement. Thanks!
Hi, Timestream databases in general look a little strange compared to traditional databases but they are great for fast and massive IoT data ingest. the challenges with retrieving data compared to traditional databases are still an issue
Good video! Had problems with my IOT device as its JSON format was different than what you presented in the video. Eventually stumbled on across the correct rule commands to send to Timestream table correctly. Pg199 in AWS IoT Core Developer guide holds the answer. Thanks
@@SteveOnIoT the Format of the JSON. Mine was like this { "state": { "reported": { "FiveMins": 160, "device_id": 9654006.1, "ModStat": 0, "CJC00": 33.8, "Ch00": 3276.8, "Ch01": 3276.8, "Ch02": 6553, "Ch03": 3276.8, "Ch04": 3276.8, "Ch05": 3276.8, "SysUpTime": 10.58 } } } Correct Rule was: SELECT state.reported.FiveMins as Five_Mins, state.reported.SysUpTime as SysUpTime, state.reported.ModStat as Mod_Stat, state.reported.CJC00 as CJC00, state.reported.Ch00 as Ch00, state.reported.Ch01 as Ch01, state.reported.Ch02 as Ch02, state.reported.Ch03 as Ch03, state.reported.Ch04 as Ch04, state.reported.Ch05 as Ch05, FROM '$aws/things/XXXX/shadow/update'
Great video, unfortunately you too are ingesting the data into timestream as single values (separate row for temperature and humidity), according to AWS there is also multi-record values supported, but i imagine it can happen only using a Lambda function and not IoTCore Rule yet, right?
@@SteveOnIoT For example, in your video sensor_5's data has been injected to the DB in 2 separate rows instead of one with both values (humidity and temperature) in the same row
@@panagiotisgiannelos3767 You can certainly transform them in Lambda but as you know multi-measure records were announced at re:invent in 2021 but I don't see them implemented. I suspect the Timestream team is fairly sparse as very little new features or material has been released since Timestream came out, and Timestream was already a very delayed release from announcement. If Timestream wanted to release this feature it would be very easy to implement, a simple button option to allow all variables with same incoming timestamp to reside in the same row.
You can add a UNIX timestamp in your rules query statement that will automatically be added to your incoming IoT JSON payload. Let me know if that is what you want to do. See my new videos where i add timestamps by the RQS: timestamp() AS timestamps
Thank you for your work. I have a problem: I create the rule: SELECT * FROM 'things/logo001' but it does not copy anything to the timestream my iot device has the following path: arn:xxxxxx/thing/logo001
Hi Jaun, the first thing to do is to make sure your IoT payload is actually showing up my using the 'monitor' blade in AWS IoT Core. After that make sure the payload is getting into Lambda by outputting the event object (your IoT payload) from a print or console.log depending on which programming environment you are using. If both of these tests are successful it is very likely you are missing the needed permission to access timestream from Lambda in which case go to IAM and make sure you have correctly given all the needed Lambda permissions as discussed in the video,
@@SteveOnIoT Thank you very much for your answer. I attach a screenshot of the shadow of my iot object: i.postimg.cc/fyXvXtrq/sombra.png The variables change from 00 to 01 without problem. I would like to save all those changes that occur in the three variables (and the date) in the timestream database but I think I am not writing the rule correctly
Awesome! I was struggling to understand how different measurements are organised after ingested to Timestreams; but thanks to you I have understood it. I actually thought I was doing something wrong and I was having this different rows for each measurement. Thanks!
Hi, Timestream databases in general look a little strange compared to traditional databases but they are great for fast and massive IoT data ingest. the challenges with retrieving data compared to traditional databases are still an issue
Good video! Had problems with my IOT device as its JSON format was different than what you presented in the video. Eventually stumbled on across the correct rule commands to send to Timestream table correctly. Pg199 in AWS IoT Core Developer guide holds the answer. Thanks
Are you referring to naming JSON variable from your MQTT topic?
@@SteveOnIoT the Format of the JSON. Mine was like this {
"state": {
"reported": {
"FiveMins": 160,
"device_id": 9654006.1,
"ModStat": 0,
"CJC00": 33.8,
"Ch00": 3276.8,
"Ch01": 3276.8,
"Ch02": 6553,
"Ch03": 3276.8,
"Ch04": 3276.8,
"Ch05": 3276.8,
"SysUpTime": 10.58
}
}
}
Correct Rule was:
SELECT state.reported.FiveMins as Five_Mins, state.reported.SysUpTime as SysUpTime, state.reported.ModStat as Mod_Stat, state.reported.CJC00 as CJC00, state.reported.Ch00 as Ch00, state.reported.Ch01 as Ch01, state.reported.Ch02 as Ch02, state.reported.Ch03 as Ch03, state.reported.Ch04 as Ch04, state.reported.Ch05 as Ch05, FROM '$aws/things/XXXX/shadow/update'
@@EvGuy Yes using nested JSON will complicate things, same issue when manipulating nested JSON in lambda ETL functions.
Great video, unfortunately you too are ingesting the data into timestream as single values (separate row for temperature and humidity), according to AWS there is also multi-record values supported, but i imagine it can happen only using a Lambda function and not IoTCore Rule yet, right?
Are you talking about batching IoT data?
@@SteveOnIoT For example, in your video sensor_5's data has been injected to the DB in 2 separate rows instead of one with both values (humidity and temperature) in the same row
@@panagiotisgiannelos3767 You can certainly transform them in Lambda but as you know multi-measure records were announced at re:invent in 2021 but I don't see them implemented. I suspect the Timestream team is fairly sparse as very little new features or material has been released since Timestream came out, and Timestream was already a very delayed release from announcement. If Timestream wanted to release this feature it would be very easy to implement, a simple button option to allow all variables with same incoming timestamp to reside in the same row.
Thanks for your awesome video. Make timestream life so much easier.
Good video, how could the timestamp be changed in the rules ?, for example, if I want to pass the timestamp that I want it to record
You can add a UNIX timestamp in your rules query statement that will automatically be added to your incoming IoT JSON payload. Let me know if that is what you want to do.
See my new videos where i add timestamps by the RQS:
timestamp() AS timestamps
Thank you for your work. I have a problem: I create the rule: SELECT * FROM 'things/logo001' but it does not copy anything to the timestream my iot device has the following path: arn:xxxxxx/thing/logo001
Hi Jaun, the first thing to do is to make sure your IoT payload is actually showing up my using the 'monitor' blade in AWS IoT Core. After that make sure the payload is getting into Lambda by outputting the event object (your IoT payload) from a print or console.log depending on which programming environment you are using. If both of these tests are successful it is very likely you are missing the needed permission to access timestream from Lambda in which case go to IAM and make sure you have correctly given all the needed Lambda permissions as discussed in the video,
@@SteveOnIoT Thank you very much for your answer. I attach a screenshot of the shadow of my iot object: i.postimg.cc/fyXvXtrq/sombra.png
The variables change from 00 to 01 without problem. I would like to save all those changes that occur in the three variables (and the date) in the timestream database but I think I am not writing the rule correctly