ฝัง
- เผยแพร่เมื่อ 3 พ.ค. 2024
- In many projects (Arduino IDE), we need accurate local time. With our ESP chips, we have some libraries to get NTP time, as I showed in video #67. Most of these libraries do not adjust for daylight-saving or summertime. Today we will change that by using standard ESP functionality. And we will see if Hippies were involved in this matter.
I am a proud Patreon of GreatScott!, Electroboom, Electronoobs, EEVblog, and others.
Links:
github.com/SensorsIot/NTP-tim...
github.com/SensorsIot/DCF77-T...
ESP32 board used: s.click.aliexpress.com/e/41caKINw
Ferrite Rod: s.click.aliexpress.com/e/_DmX...
#CommissionsEarned: The links above usually are affiliate links which support the channel (no additional cost for you).
Supporting Material and Blog Page: www.sensorsiot.org
Github: www.github.com/sensorsiot
My Patreon Page: / andreasspiess
My Bitcoin address: 19FSmqbBzb5zsYB1d8Bq4KbxVmezToDNTV
If you want to support the channel, please use the links below to start your shopping. No additional charges for you, but I get a commission (of your purchases the next 24 hours) to buy new stuff for the channel
For Banggood bit.ly/2jAQEf4
For AliExpress: bit.ly/2B0yTLL
For ebay.com: ebay.to/2DuYXBp
profile.php?...
/ spiessa
www.instructables.com/member/...
Please do not try to Email me or invite me on LinkedIn. These communication channels are reserved for my primary job - วิทยาศาสตร์และเทคโนโลยี
Just for the record, you would NOT WANT the time servers to implement daylight savings time or other features. It is totally correct to have a global UTC time network, and then for local presentation do the work to implement time zones and daylight savings. It is not a weakness that NTP only uses UTC, it is a feature.
And no leap second smearing either :)
I would have liked a function call in addition to GMT where you can get local time...
@@AndreasSpiess it becomes useless the moment you want more then a signal to set the clock its just to make all the clocks on the WAN all beat at the same time. if you have to add more calculation to it to get the local time or even daylight savings from the server you add milliseconds to the sync pulse thus you create a desync. the whole reasons to have a distributed network of ntp server pools is that you have a server with the lowest latency. if for instance you would only have a ntp server in Hawaii and the whole world would have to sync with it. the pure latency on the other side of the world would trow the whole network off.
@@AndreasSpiess If you have mqtt in your project, then fetch the timezone string from there.
Or use one of the "private use" options in DHCP, option number 224-254
@@var67 IP ?
Btw, "LSD was first synthesized on November 16, 1938 by Swiss chemist Albert Hofmann at the Sandoz Laboratories in Basel, Switzerland".
another guy with the swiss accent changed the world, without youtube.
Of course I know that. I once worked for Sandoz. It is located 20km from here. But I cannot always mention that a Swiss invented it ;-)
@@AndreasSpiess apropos Sandoz: some years (or decades) ago, a read the books of Roland E. Best, who was with Sondoz These days. Very fine and helpful books! Was he a colleague of you?
It is wonderful how you dig into things that I find useful, do all the work of finding out the details and then present it all nicely wrapped up so I can use it almost without any hard thinking! Thanks!
That is my part of the deal. You did your part in writing this comment. Thank you very much!
Andreas, your videos are AWESOME! I'm an electronics / home automation hobbyist and you have already solved many challenges I have run into. You are my favorite TH-cam publisher hands down. Thank You!
Thank you! I am glad you like my channel.
As usual, an excellent contribution to the maker world. Perfect level of detail and information. I had given up with NTP time servers and just poll my local Raspberry Pi as this seem to deal with the daylight saving issues.
How do you poll it from the Raspberry?
Good introduction on the timezone implementation for the ESP and NTP server. Now the fun part...
Lucy in the Skyyyy with Diamonds :P Half of the structure POSIX starts from number 1 and the other starts from 0, maybe some was with LSD and the other with magic mushrooms. UTC grants timming for all purposes and host needs to implement DST in its SW: It can be done using automatic using Wi-Fi, GPRS , GNSS ... to acquire the timezone or input manually in the region settings, and if has timezones incorporated.
Thank you for making another interesting and informative video Andreas. Computer time is a complex subject. I am often working nights over the clock change and it is always a concern whether any new system among the several thousand servers we have will throw up errors. At least this year we are spared the leap second addition. Mind you it is always depressing to watch the clock roll back and realise you have to work that hour all over again.
There is still hope that we will stop this nonsense ;-)
Useful and interesting, great video on handling time on an ESP. Thanks for sharing
Glad you enjoyed it!
Thanks for a very informative (as usual!) video. I made a clock a while ago which ran off NTP but couldn't work out how to easily make it automatically change in response to DST so I cheated and added a switch! I'll have to revisit my code now.
Should be simple. But never change a running SW ;-)
@@AndreasSpiess That's wise thoughts from a wise man :-)
Thanks for the insights in time handling. I made good experience using the ntp Server in my router instead of creating Internet traffic and CO2 emissions. All of my ESPs and most other devices have the same time reference - my router. Best sync, and avoiding traffic.
Good idea. The Adaption to local time might still be necessary.
@@AndreasSpiess Yes indeed, it is. The router does not make any conversion. My IP list contains > 60 Units. So using router (Fritzbox) as NTP server helps a lot avoiding traffic.
@@klassichd10 yes, it is not considered "polite" to have all the clients on a network poll a remote NTP-server. You are, as you do, supposed to query the local timeserver.
I wonder how many systems honour the NTP-server option from DHCP, but it should be fairly easy to implement in standard routers.
@@LeifNelandDk
Windows didn't in the past... Would just use NIST or whatever the other default was regardless.
I ended up setting up a firewall redirect on my home LAN, and a fake DNS entry for time.nist.gov at work. In both instances, they pointed to the local NTP server.
Needed this tutorial, thanks a lot.
You are welcome!
Great video as always
Great project👍
Thanks for sharing👍😀
You are welcome! Many comments suggest it was interesting for many viewers.
It was both useful and interesting, as always! I did want to point out that up to 3 NTP time servers can be specified in the "configTime" function, separated by commas.
Good to know. Thank you!
I'm a simple man, I run everything on UTC, then adjusting offsets when aggregating or displaying datetimes
Nothing different here, I think. Just already written by somebody else and ready to use...
That is how you actually use time and date calculation properly.
Java has JUST start getting time and date calculation right. And one can't call Java a young language...
A very interesting and useful video, thank you Andreas!
You are welcome!
Perfect. I created a solution for my projects but this is much cleaner and simpler.
:-)
Good video on an important topic. Glad you fixed the example on Github. It's OK to call time() as often as you like the SNTP library used by the ESP Arduino core only calls it once an hour. Another thing that is useful to know is that if you have a well configured router that issues NTP servers in its DHCP and/or DHCPv6 responses your time server will be automatically configured. This will override any in your sketch unless you take steps to turn this behaviour off (a bad idea normally).
Thank you for the info. I assume it still is I’m GMT and you still need the time zone adjustment?
@@AndreasSpiess Yep. The SNTP client functions exactly the same. You need to call configTime() to specify the local timezone and DST offset and enable the SNTP client. You can continue to specify servers here too but they will be overridden by whatever is in the DHCP response. I find this very helpful in reducing the configuration information that needs to be set up and stored on each device.
A quick note from a NTP pool server operator. You do not want to excessively poll public NTP servers. Most implement some sort of blacklisting if a client constantly demands time from them. Ideally, you should ask for time quite often in the beginning to sync your own local clock, and then back off as your confidence in your local clock increases. If you look at the “ntpq -p” commands output on a Linux box running NTP, you’ll see this behavior quite clearly. As time goes on, the time to the next query increases. The reason we’re so stingy with bandwidth is that when you’re already giving away time for free, we’d like to give it away to as many people as possible.
Also, good work that you implemented the DNS-name for the pool instead of just the IP of some random server, that’s very important since the amount of available servers can vary very much, here in Finland for example we have something between 17 and 50 NTP servers in the pool at any time...
Thank you for the info. We have indications that the function time() only calls the NTP server once an hour. Could you check and confirm that with your knowledge? Then we would not need any manual throttling.
Has the number of requests per hour increased steeply thanks to IoT devices?
@@AndreasSpiess I'm not close to a computer with the Arduino IDE installed, and the time.h library seems to be very hard to find online for some reason. However, every reference I can find seems to have a parameter in the library that sets the update timer, usually between 1 and 5 minutes. A normal NTP client (without ibrust och burst set) will start off trying to poll the server every 64 seconds, and will slowly increase this interval as it gets more into sync, eventually maxing out at 1024 seconds. So any device that has a polling interval of more than 500-ish seconds is in my point of view "in compliance" with the not so clearly stated NTP Pool guidelines. Setting the polling interval to 3600 seconds should be more than enough for any device that need only 100 ms precision, as NTP is designed to give precision in the millisecond resolution with a polling interval of 1024. If you need better then that, there's always PTP. :)
@@NiHaoMike64 It's rather hard, as the pool load balances all traffic to the pool servers depending on physical location of the client, as well as a factor that you set for your server. So mine see around 70 kbps constantly, but I'm in a quite "low server population" area, with a rather high "client population". The better IoT vendors usually follow the NTP pool guidelines and request a separate "vendor name" in the pool, and send all their clients there, so that they can direct their clients better towards pool servers provided by them. But IoT has definitely had quite a large impact, since there's always from time to time some vendor that gets raked over the coals for polling time every second from the general pool. :)
thanks Andreas, ive been using NTP in most of my esp8266 and esp32 projects, but i never knew about the TZ_info string...i always just used it as "GMT+10" and changed it when daylight savings started or ends... but the complete string with daylight savings start and end dates seems alot better..
i also tend to load the NTP time into the inbuilt clock and only update it once every 24hrs (or if the micro reboots)
We do some investigations. It seems, that the time function also calls only every hour the time server. So we would not need the second function.
@@AndreasSpiess you may want to look into the functions, setSyncProvider(getNTPtime) and setSyncInterval(86400); ...they are built into the time library and are usable with ESP's
The tm time structure came from UNIX which was from Bell Labs. The months and weekday values start at zero to facilitate retrieval of the names which are stored in string arrays and arrays in C are always 0 based for the index. In this way you can use these values directly as the index to the arrays of the months or weekdays without any additional work.
Good to know. Thank you!
Also
Doing arithmetic is much simpler
E.g.
You start with a Tuesday, which has code 2
13 days later .... what weekday is it?
simple: ( 2 + 13 ) % 7 = 15 % 7 = 1
So it is a Monday, since Monday has code 1
In fact it is most of the time quite the opposite. The real question should be: Why the heck do you insist on 1 based indices? They just make life more complicated.
Yep and IF they had started with 1, some idiot (tm) would try to set it to 0.
I figured out that there are also problems with the configTime() function. It works asynchronously and gets the NTP time, however:
- During the period it gets the time it sets the ESP32 time to zero!
- You don’t know when it is finished.
- Existing app variables e.g. service startup time must be adjusted
to have proper time statistics after time changes.
I did the following to overcome this. I save the current UTC time and count in parallel seconds while calling configTime(). After two seconds I verify if the time() changed by more than 3 seconds, if yes the new NTP time has been set, otherwise I restore the clock to the value before calling configTime() plus the seconds since then. Existing variables of my app like startup time of my services, etc. I adjust.
Thank you for the great videos, now I understand even more of the the TZ magic string.
Regards from Hannover, Germany.
I try to check the year to find out when the NTP time arrived.
Concerning the time() function: A Patreon with an own NTP server found out that the ESP only calls the NTP server every hour. In between it keeps the time with the internal counter. So you might not compare real NTP time if you call time(). But otherwise it is a good idea to compare two different sources to find "the truth"
@@AndreasSpiess I am using a separate ESP32 64.bit to track the time independent of the time() function, for this case.
My software updates the time once a week via NTP, I did not know that the ESP32 is doing it automatically. I will investigate into it further.
Watching it during unix 1,714,253,105.
Never gets old.
???
2:34 tempus fugit
This is so cool, I'm not sure where you find your inspiration! 😉
From problems I have to solve ;-)
Merci Andreas. I like this vidéo with ESP.
You are welcome!
Great video, thanks!! I found a library that worked well, but would have liked this a month ago :) as it seems simpler... That said, an issue I had with NTP was connection reliability. For some reason, I would often get no response from the NTP servers (or some error i couldn't figure out) Wasn't hitting them often, and even tried setting up my own NTP server. Didn't cause much of an issue until when the ESP restarts, it needs the time ASAP, and would of course display the wrong time until it could get a response. Instead, I now use MQTT to get the Time from a NodeRed machine that responds and send the UTC code back on another topic. Another option I thought of was to fetch a simple web page and grab the time from the headers, or again, have the http respond with json that contained the details i need.
My sketch checks if the year is ok to decide whether data arrived from the server. But using MQTT is also a good way.
Thank you , it's very helpful
Glad to hear that
gotta say the decoding of the time string was the best part of the video :-)
Thank you!
Thank you! A lot of useful information. Some comments:
time() function, if the initial connection to NTP server is successful, provides the correct result even Internet is not accessible. In other words- looks like the GetTimeReducedTraffic() idea is implemented INSIDE this function and therefore GetTimeReducedTraffic function() is useless. Another conclusion- time() cannot be used as an indicator of Internet presence
You are right, time() runs off the internal clock if not connected to the internet. However, this is not very precise (for a Swiss ;-) )
Thank you for this video. Filled a few gaps in my misunderstanding 😜
You are welcome!
Andreas, your videos are always a thumbs up, because you provide a well planned video.
You include a coarse level overview and then give a level 2 and 3 view of the information. Thank you for using Arduino code.
**I think you have a verbal "typo" at **5:12**, you say "at one o'clock in the night", but the value shown is **2:00:00**.**
Which is correct?
One is in UTC
you are my favourite teacher
:-)
tm_mon and tm_wday conveniently begin at 0 for efficiently indexing into an array of strings to get the printable name. tm_mday starts at 1 because it isn't generally used as an array index.
Aha. Thank you for the explanation.
As far as I recall, struct tm was present in Bell Labs 7th edition Unix, so it came from New Jersey, not California. There were possibly still hippies involved, though.
I knew a few of my viewers have more details. Let’s wait till the Californians comment...
From California -- I cannot chime in on whether the origin was California or New Jersey, but I can say that in might have been a botanical rather than LSD regardless of the location of origin
Surely you've seen Dennis Ritchie's beard.
Don't underestimate the power of a having a "system/localtime" MQTT topic for the devices to subscribe to. :)
Good idea!
Vielen danke, Andreas.
I've created a ntp clock on an oled screen, but had some trouble with the date, as it was American with the month first, but as a Dane, that's just silly to me.
I'll try your code, and maybe just update every minute, to not call the server too often.
Anyway, thanks again.
No problem with the sever calls. We found out that time() only calls the server every hour.
Bravo, la structure tm a toujours été pour moi un véritable casse-tête et je suis sûr que vous avez trouvé la véritable raison de ce problème : Unix + beer + canabis + LSD (+ easy girls) ===> struct tm !!
Le casse-tête peut être encore plus facile si tu bois une bière toi-même.
Fabulous!!! Just recently we had switched to winter Time and my Code did not know how to adjust itself.... Embaresement in front of wife :) now you saved me :)
So you are her hero from now on ;-) In this case you do not need to mention the source of your knowledge...
I now get my time/date from the 'date' section of a web-page header. The advantages are that it is just a simple call to any web page, so no restrictions on how often you call it, (its just a web page). DST is already calculated for you, (the timezone + offset is written at the end) and it's a lot less code! (No extra time libraries required!)
Parsing the info from a single 'date' string is pretty simple to do and gives you everything you need....and if you use a well-known search engine for the header info then you also have the advantage of 'leap smearing'....they use their own public NTP server. (I was under the impression this is also a Stratum 1 server but I'm struggling to find that info now.)
I adjusted the code from the link below to give me just the 'date' info:
techtutorialsx.com/2018/06/08/esp8266-arduino-getting-http-response-headers/
Interesting idea... Thank you.
Michael Graff is quite correct. Use UTC time across the system, this avoids data logging complications when local clocks go back an hour. Then adjust the time when presenting to users at the point of consumption. Thus the locale of the source device can usually be inferred by user context. (The user can say in which timezone the data should be presented for any given device.)
I think (hope) this is standard in all applications where you expect this issue. I remember the time where this created headaches...
I think the term "tm" came from the phrase "time management." It was during the beer and LSD era when programmers thought they would show off their alleged creativity and muck up the facts enough to make up stuff like a lot of the programmers did. This was the time of Fortran, BASIC...etc programming languages. Like everything else in programming programmers must have created names for their language using strange names from the local phone book, variations of languages such as pig Latin, olde English and French. There's my explanation and I drank no beer or dropped acid to arrive at my conclusions. I probably had yogurt earlier to clear my head and digestive tract. Pretty straight forward wouldn't you say??? 😃😄👍🍵☕🍕 Thanks for being a great teacher Andreas..
And maybe they had valid reasons and nobody remembers, why... Their resources were very limited back then.
That's a good presentation, but you certainly like doing things the hard way. I personally would have used a GPS module with the ESP32 and saved complexity and myself from writing a ton of code. A GPS based clock would be totally self adjusting. I'm sure you could have figured it out pretty easy. Too bad you didn't grow up with me during the 80's in South Carolina. We would have been drinking, fighting, going to wild parties and building computers every weekend. I hope you have a great evening.
My lab is in the basement. So GPS is not a good solution for me :-(
VMS uses an epoch starting on 00:00:00 of November 17, 1858 (base date of the U.S. Naval Observatory's ephemerides)
Some OSes use - Modified Julian Day zero - May 23, 1968 as epoch start
Macs start in 1904
Windows NT and COBOL uses 1 January 1601 - (the first year of the 400-year Gregorian calendar cycle)
whilst some other systems use 5 October 1582 - the actual date of the transition from Julian to Gregorian calendars
"Standards are wonderful, there are so many to choose from"
Thank you for this interesting info. I did not know that there are several such standards! Now I know I have to be cautious.
The weird numbering was standard in the US. We built Stirling cryogenerators which were also used in high end (super) computers. They invariably counted 4 cilinders as number 0,1, 2 and 3...
Interesting! So you love complex numbering systems. We Europeans also need a much easier system for distances etc ;-)
@@AndreasSpiess No I don't. Like you I am European (Dutch) and prefer our metric system starting with a one.
It is worth noting that any machine in any network can act as an NTP service - type "man ntp.conf" on a *NIX machine. This is even the recommended way of setting a time reference for any local "organization" to avoid stressing the internet time servers... as well as the querying IP address being banned/blocked :-D. So any machine with a low enough stratum number can be setup as an NTP server, which can be synchronized with internet time servers using NTP... or using a mere GPS when no internet connection is available.
Thank you for the comment. I did not know that fact.
@@AndreasSpiess It's my pleasure :-).
Great video again. Just one remark. Update NTP is only with configtime function. getlocaltime_r() in getNTPtime example code does not update ntp in my test. Did 24h test with example code clock was 2min behind. Restart updated time correct again. So update is done only when configtime is run and you have wifi of course. found nice extra (esp32)configTzTime(TZ_INFO,NTP_SERVER,NTP_SERVER2,NTP_SERVER3);//this updates via ntp and posix is integrated.
Maybe the example code has an error. But it should be easy to debug...
Finally a way to synchronize DCF clocks in Australia. 😎
P.S. I'd check the drift between NTP server requests and keep doubling until it becomes significant... Limited to about once a day, (32 hours) maximum.
We do investigations. It looks that time() does not call the server every time. Only every hour. Which would make the second function obsolete...
Great Video as always Andreas. Is there any documentation for esp commands "configTime" and time()?
I did not find something ready made. Time() is a unix function.
Interesting video, as always!
"tm" never seemed that weird to me, to be fair. I've extensively used that structure back in my network/server/socket programming days. The only field from that structure that I always found "weird" was "mday", which starts at 1 instead of 0 like all others. (Fine, year starts at 1900, but that's basically 0 from a "century" point of view).
Minor correction about Aknik's code: His variable names are most likely in Italian (or what looks like Italian to me). Or at least 100% not Spanish. That much I can tell.
I think it is no problem if you know the definitions of the structure. I searched for my mistakes when I first got the results :-(
Concerning Italian: Obviously I mixed them. Thank you.
Agree too. To me it absolutely looks like Italian. As you say, not Spanish for sure.
TimeT has already been converted to 64 bit in some unix libraries. So 2038 won't be the end of time, at least not for systems running on 64 bit processors.
Good to know. Thanks!
Y292277026.596K
I got my own in Iceland running now! \o/
Finding and sorting out the hardware problems took some effort without an oscilloscope and the resistor values are different, when your coil has 198millihenry... ;)
Congratulations!
BTW: most time zone strings are defined by "#include ", at least in the ESP8266 libraries (I have not looked into ESP32's yet).
Thank you for the tip!
It's about time
:-))
Another trick to get the NTP time into the IoT devices is to use the main router as the NTP server within your local network. For example Fritzbox can do that.
What are the advantages to do so?
- security: the devices don't need to have full internet access (like in my case I have all my smarthome devices blocked from accessing the internet) but they can still get the acurate time all the time and immediately from the router directly
- availability: assuming that the router always has the latest acurate timing received from a public NTP server it can provide that time always to any of the connected devices without needing all of them run public NTP requests at all
- proper synchronization of all smarthome devices: if the router would have the wrong time, then ALL connected devices work with the same time and don't run asynchronously
The Fritzbox allows a setting for how often it should run the public NTP request => no danger for any connected devices to overshoot the request limits and getting blacklisted.
It works very well with whatever kind of device in my case. ESP8266, ESP32, Raspberry, Android, Linux, Windows. Even Tasmota runs perfectly with this method.
It is amazingly simple.
In the Fritzbox you define the public NTP server plus the frequency for updating from it.
And you activate the function of Fritzbox becoming the local network NTP server by symply clicking the related checkbox.
The local NTP address is (with Fritzbox routers): fritz.box
That is the name for the NTP server you must enter in the NTP settings of each device.
As simple as that.
No further hassle with any additional settings.
Whenever a smarthome device boots up and connects with the router, it will immediately receive the correct time right away - even without being connected to the internet.
Works all fine for me (I am a bit paranoid when it comes to internet security)
Thank you for the tip!
Great Video ! Thank You. Can I know whether it is possible to store time when the ESP32 goes to deep sleep and retrieve it without getting NTP time ?
The ESP32 has RTC memory which keeps its content during deep sleep. But your clock will be the same if the ESP comes back from sleep. So you would have to correct the time.
Hi Andreas.
I Love this channel. find it's my 'goto' for all things ESP.
At the start of your listing, you state 'getTimeReducedTraffic(int ms) where ms is the the time between two physical NTP server calls' should this be seconds?, And are the lines ' 'getTimeReducedTraffic(3600)' and 'getNTPtime(10);' mutualy exclusive.
I thought UTC had its roots in astronomy and was agreed to be Jan 1, 1900.
Yes, the two lines are mutually exclusive. And if you go there, you will see that the second one became obsolete.
@@AndreasSpiess Thanks for the confirmation & update.
Good video, can you please provide de diagram of the DCF77 antenna, resistors values, etc... Tanks
Maybe you watch the video I mentioned?
I am ashamed, as a 61 year old physicist, to say I'd never heard of Unix time. Thank you for opening my eyes.
We have a similar age and I also did not know it until recently ;-)
Andreas, no beer or drugs were involved with "struct tm". The ISO 8601 standard that establishes Monday as the first day of the week was first published in 1988 almost two decades after "struct tm" was defined. In the United States, where the code was written, Sunday is considered the first day of the week (again pre- ISO 8601). As you are aware, the C programming language uses zero-based arrays thus the tm_mon and tm_wday can be (are) used to index arrays for converting to character (string) representation as in strftime(). January and Sunday therefore have zero values in tm_mon and tm_wday respectively.
Thank you for your explanation!
Yes Andreas, for us it's normal: Monday is day 1 in the week. But over the big pond they thinkl different about that. My Canadian Colleague told me, he can't believe, that Sunday could not be the first day in the week. I have no glue, why they decide to this counts. Also in the Bible, sunday was the last day of the first week...
So, i agree, beer or much stronger things causes this weired situation.
They usually have Sunday as 1 AFAIK. Not 0
@@AndreasSpiess your right, more confusion happend ;-)
WARNING: You should only poll a NTP server every 64 seconds at most frequent, backing off to hourly once you're happy with local clock stability
Polling every second is classified as abuse and can get you filtered
You should also run a NTP server in your local network and sync everything inside to that instead of the pool servers
You are right, of course. This is why I mention that calling it every hour is sufficient (9:28). But I did not warn :-(
Hello Andreas, just for the records :Starting at 4:50 you state Switzerland would be behind UTC, as far as I know Switzerland is one hour AHEAD of UTC (at least it was true until now, don't know what will happen if the EU stops that stupid winter/summertime switching one day), not behind as you mentioned in the video - might be different for the area of Bern but this is a different story :D. So the correct term would be UTC+1h = CET (aka MEZ) and UTC+2h = CEST (aka MESZ). Same for Germany ... However the definition string you mention is quite simple to understand and defines the timezone CET to be one hour ahead of UTC (reverse logic !) meaning CET-1 is what you need to get UTC ( CET -1h = UTC) and CEST-2 (CEST - 2h = UTC) is again UTC ... Switching time in March (M3 at 2:00) and October (M10 at 3:00), every time in the "5th week" (= in reality last week of the month). As usual the challenge is to not get into a recursive loop while switching back from summer to winter time (as you switch back at 3:00 to 2:00 and will have a next 3:00 the same day after doing that) ...
You are right. My explanation is wrong. At least the function should work ;-)
Looks great, I was also searching for something like this for a project. My only concern is: As the EU is going to abandon DST at some point in the future, this implementation will produce wrong time values from there on. How can I take precautions for that? I thought about a web interface toggle to change the string between no DST - normal DST - permanent DST. Thanks!
I do not plan for more than 10 years ahead ;-)
IIRC There was a period before the early 90's where the UKs GMT/BST switchover dates were defined in a less computer friendly way than they are currently (almost as if you can't rely on politicians to make your life easier). instead of last sunday it was the sunday after the 4th saturday. un*x sysadmins had to install a patch when we switched over to 'always last Sunday'
Interesting. Maybe the next patch will become necessary when these changes stop ;-)
Great video!!
But how do i make the antenna??
I showed the video where I built a transmitter.
NTP I remember using GPS to get may time from some caesium and rubidium clocks. Would you like to see my gold for the day Andreas? Nice piece this time.
Congratulations! I get really jealous!
@@AndreasSpiess No you don't you have youtube mate. By the way just cleaned a button I founded close to it. British staff officer 1788 first fleet.
What is the impact on power consumption when using time functions or are they enabled by default?
I assume it is only there with the time.h library. Otherwise it would not know where to call...
They should be pretty minimal. When you enable SNTP by calling the configTime() function the SNTP client will only call the server once an hour after the initial synchronisation. It uses the very efficient timer mechanism within the LwIP library used by the ESP family for their TCP/IP stack. There are no additional threads for example. There will be a bit more memory usage but not much. The code for SNTP is pretty much always there whether you use it or not. I've been fixing/updating SNTP support in the ESP32 and ESP8266 Arduino libraries recently which is why I come to know all this...;)
Oh no! Time!!! Einstein himself would have an apoplexy trying to deal with programming time! Since working on a system recording real time data across many time zones a number of years ago, I became convinced that time really doesn't exist! Lol! So go on - timestamp your data and see how many variables you have to include to separate out or synchronise each piece of data (and don't forget the duplicated data that occurs when the clocks goes back at different times in different time zones!)!
Fortunately I usually do not go below 1 sec precision in one time zone. This makes it a little simpler...
Its an insanity made by silly humans unfortunately. UTC or GMT is the way out but it apparently is still to hard for the normal's out there. There is absolutely no reason for the time changes any longer this is 2019 going on 2020 not 1600's where perhaps farming was tied to day light hours.
Its a tireless legacy that needs to just go away, but if you need to do these calculations you would do them on your backend system. Fortunately for me I use Java mostly so Joda Time takes care of this in 3 lines of Java.
It was a good fix for those that need to do this on the embedded side, there is more info on how to adjust for drift using the pool of ntp servers on pool.ntp.org another way to get time if you have an iot out there with a GPS is to sync time with that as GPS time has to be accurate, this is also an epoch and works very nicely with the rtc library for most clocks.
@@sethrd999 unfortunately GPS time does not match UTC either, and while the offset is trivial, your original data is always the reference, not the backend calculations, so if you record GPS and convert to UTC, you now may have to hold the history of calculations that determine the offset between the two at the time the data was recorded. The more critical a timestamp is to your data, the more time itself starts to look like an amorphous beast!
@@sonovoxx You're right, and having had to deal with timestamps in the past similar to what you have, I second your concerns about how complex temporal calculations really are, especially when we live on a wobbly ball of dirt and water that doesn't even have a well-behaved rotation! Regardless of this, let's also not forget that POSIX time is not the same as UTC, nor is it GPS time, nor TAI. It is close but not quite UT1 either. Furthermore, the time_t value used with the header (not ) in C and C++ is not necessarily the POSIX time value, although I have yet to see an implementation that didn't use it. Most modern libraries also have switched to 64-bit time_t a long time ago by now, but of course the exception is most likely in this very field of micro controllers that we are dealing with on this channel. :-)
@@benhetland576 stop right now... you're bringing back all the nightmares! 🙉🙉🙉😂
Got my first radio controlled watch. Got my esp32, now wondering can I use just wire as an antenna if I wrap it around my watch?
This should work. Maybe a few windings...
4:51 - actually we are 1 or 2 hours ahead of UTC. These numbers mean UTC is 1 or 2 hours behind local time.
In a UTC time string CET & CEST are denoted by the Thh:mm:ss.ms+1:00 and Thh:mm:ss.ms+2:00 time suffix.
Thank you for clarification.
Hi Andreas, thanks as always! It's something wrong with the DCF77 link to github?
I will check it later.
Works now. Repo was private. Sorry...
@@AndreasSpiess Perfect! It's working now! Thanks again.
Nice video, many thanks. May be I haven't understood the main reason for using NTP server with ESP32 for time. So please allow me to ask: Why you didn't choose to use DCF77 directly? You don't have to worry about summer and winter time correction or the number of time you send request to NTP. With DCF77 you always have the right date and time. Is the main reason for your selecting the NTPserver is the ease of adapting the NTP structure and available ESP32 functions in the sketch? Please comment. Thanks
I thought I mentioned the reason: I do not get DCF77 reception in my basement :-(
Thanks for the video. One question.... why are you using time.h instead of TimeLab.h (which is Arduino's time module)
I do no longer remember why :-(
Andreas did you ever used LiFePo4 batteries with Lolin D32 (ESP32)? It works as well as LiPo batteries?
LiFepo batteries must not used instead of LiPo batteries. They are not compatible. LiFePo batteries have to be charged with 3.6 volts, not 4.2
Thank
You are welcome!
@@AndreasSpiess I love you for being so nice to random strangers on the internet!
I try to answer all comments.
I just went through the same exercise for an ESP8266 project, and eventually came up with pretty much the same solution as presented here, but with one minor difference: after the setenv(...) call, the examples I referenced included a call to tzset(). I'm not sure what this does, or if it is needed, but it hasn't hurt my code...
I also saw this command. But it seems to work without ;-)
good video.
but isnt using time as described as in the "simpletime" example in arduino for esp32 much simpler?
i use a slightly modified even shorter version of this example and it works well and is less code and more readable.
Are you sure your example automatically changes times in spring and in autumn? This is the point of this video.
Year 2038 is only for signed 31 bit systems. Declare time_t as uint32 and have no problems until 2106. No problems except when you talk to the 31-bit computers ;)
Gerard Sharp no, int is signed 32 bit so you'll go in to negative values past 2038.
Well, as per the applicable standards, the time_t is proclaimed to be of "some signed integral type", and some of the API functions return for instance (time_t)-1 as an error indicator. So even though the type is supposed to be able to represent times before 1970, the exact times 1969-12-31T23:59:59Z and 1970-01-01T00:00:00Z can be problematic or ambiguous; the former because it can actually be an error, and the latter because 0 is often used as a default initial value or to indicate other conditions such as "no time" or "missing time data".
Times prior to 1970 need to be represented... Not often, but it does happen.
Last thing I knew before this was sending NTP requests manually via UDP and converting the epoch reponse also manually.. that was too long 😂 ..
**let alone handling the error if the packets failed to reach the server or if we got no reply bla bla bla.. headache 🤧
Thanks
This is why my large NTP clock is built around a RPi instead of the ESP32, I wanted "proper NTP" and googled ESP32 NTP support for 5 minutes, got a headache and reached for the RPi. :)
@@sstorholm the only good thing about it is that it taught me how things are working in the background; in details.
It also might be good for small projects especially that it's cheap, you know we won't use a computer just to schedule a plant irrigation !
Guten Abend Andreas, Interessantes Video, habe gerade Gefunden! Welche Distanz ist Überbrückbar mit den DCF Sender mit der Ferrit Antenne als Sendeantenne ? Grüße aus Wien
Ein Meter oder so.
Many program's in the early days start every count with the 0...so you have more until you reach the end of the range. In a time where memory-space is expensive that was a big issue. Also indexes in programming languages start many times with 0 as first element.
That doesn't mean they must be out of their mind to used this 'golden rule to start' also on month counts.
So you are absolutly right.
I agree in general. Here I had my problems with consistency. and there was no need to start with 0 as no number reaches even 255... And they are all int.
@@AndreasSpiess I coud see a problem with uninitialized systems trying to get undefined month 0 if it was done otherwise.
@9:52 Aknik is Italian :) Buon Giorno!... Good Day!
Thank you!
Still waiting to see if you can do lifi lighting -- lights that provide internet ( this would be awsome )
Maybe one time...
Andreas, you are about 1125 KM away from Anthorn Radio Station that broadcast the UK time, and you are about 300 kM away from DCF77 source in Germany. I was wondering if you could make the test to get both signal on two separate radio controlled clock side by side. Or on the same clock that can be configured. Are you too far away (you said 2000Km for DCF77)? Is this related to curvature of earth or something blocking you? Maybe you could discuss how far it goes the way you explained Lora distance? Maybe you are not interested in UK time... Could you detect discrepancy if one goes wrong and the other stay right? That could be interesting video, at least for me. :-)
These signals are propagated through the stratosphere. This is why they are generally better during night. So the distance the signal travels is always slightly different.
@@AndreasSpiess Thanks. Being somewhere in the triangle between Anthorn (UK MSF60 at 60kHz) Frankfurt (DCF77) and Allouis in France (162kHz), I should be fine then. If ever Europe disagree on Daylight Saving, I should be able to get all of those.
Because of you, I am trying to put Ikea BURRIG Radio-controlled wall clock in my Xmas list, but this is not available Europe wide. It is available from Ikea Switzerland, but not in Belgium. I may have to do import.
Your project is very interesting, please keep us update and share your ESP32 code.
08:20 this is OK to start counting from zero ;D and Sunday is the 1st day of week because of national standards.
I have another question: why on earth they use int variables to store numbers like day or hour??? use byte variable! save some memory!
Valdis Black Because int is the native memory size and requires only one clock cycle for access, byte requires additional clock cycles to isolate from the bytes returned during memory access.
@@christopherguy1217 Maybe it would be better to say "native memory size for ESPs" as they are 32bit processors, whereas for a classic arduino, byte (= uint8_t) would be the better choice
Zeero unfortunately the number of bits in the word length of a processor does not directly define the native types available in C and C++. This is precisely why we have sizeof operator...
Perhaps it is also the Swiss that discovered time (well at least they might have been the first to commercialize it :-))
You are right. We definitely know how to make money wit watches;-)
LOL Brits will create own time zone after Brexit. Made me spray my coffee - too funny. :)
:-))
It's going to be GMT - 50 years :-(
^^^^ This!!!
Or maybe they'll just keep it as is. You know, like before EU ;)
It should be 1.61h difference, one "imperial" hour, not metric one.
Hi Andreas to change to 60khz UK is just changing line , ledcSetup(0, 77500, 8); to ledcSetup(0, 60000, 8); also the way we are going in the UK the politicians will still be discussing a Brexit deal in 2038!
I assume the coding is also different. But I did not check.
As I watch this, I'm refactoring the code produced by two days of manic hacking about half a year ago. It was written with beer also.
So good luck ;-)
Hello. Do you have a schematic for the antenna connection with the ESP32 please ?
Did you watch video #287 mentioned in the video?
👍👍👍
:-)
The reason for starting at zero might simply be that the values are calculated mod 7 or mod 12.Why bother to add 1 to the result. Moreover, finally, you willoutput day and month names. And array indexing starts at 0, too...
I do not bother adding something if I know. But when I first saw the numbers I started to search for my mistakes :-(
Sunday is the first day of the week in the US
Whether Sunday or Monday is considered the first day of the week has a very long (and somewhat muddled) history indeed-well before ISO decreed a standard in the 1970s.
In the US, Sunday is still considered the first day (in disregard of ISO). It also doesn’t matter as much here as in Europe because we don’t refer to dates using the week-of-the-year + day structure, but just stick with month + date structure.
Interesting discussion and links here on Stack Exchange: history.stackexchange.com/questions/35979/when-did-europeans-make-sunday-rather-than-saturday-the-seventh-day
Does this support the NTP Drift function?
I do not know this function.
No. The SNTP protocol (a subset of NTP really) doesn't calculate local oscillator drift. The client updates the time every hour and it will jump the system time to whatever the update says the new time is. There is no clever slewing of time drift as you get in bigger OS like Linux. It's a bit much for an embedded device like an ESP.
You mentioned video with actual build of transmitter but I cannot find link to it. Can you post a link?
Search for DCF77
How is the ferrite rod antenna connected to the ESP32? any circuitry required? I have one from a DCF77 receiver and it has a capacitor between the 2 wires, I read this is called a tuned antenna. Do I just connect the 2 wires to the ESP32? Thank you
Did you watch my video #287 shown in this video?
@@AndreasSpiess I did, but I'm not sure if I could just replace the ESP32 single wire antenna with the ferrite antenna and its capacitor, or should i take any precautions with the circuitry?
@@SEOng-gs7lj You can add a transistor if you want.
Turn on. Tune in. Drop out !!!
?..
Andreas Spiess you are not a hippy if you don’t know that slogan :)
en.m.wikipedia.org/wiki/Turn_on,_tune_in,_drop_out
Welcome back, Andreas. The first comment I did not have to unblock. Thank you for the info. The book I showed was about Leary and his crowd traveling on a bus etc. But I did not remember this slogan. Anyway, I am a little too young for a true Hippie. In 1968 I was only 11 and lived in the countryside...
Ive build a Neopixle Clock with a 60 LED circle and a esp8266 with the NTPClient Library. I assume it requests every second from the NTP Server (But its my Fritzbox). So im having alot of traffic i think. Or does the NTPClient library only update on "update();" ? I have no clue.
Automatic SNTP updates occur every hour on ESP32 from what I remember. Simply reading the time does not cause a new request.
@@tin2001 Would be nice. How can i figure it out? Turning off the WiFi and see what is happening?
@Andreas: Tin2001 is right. It was proven by a viewer.
@@AndreasSpiess thank you alot!
just wonder.
If function time() on (arduino core) time.cpp always call SNTP (need internet connection), then ESP8266 doesn't have any internal clock ( i mean counter)?
if that so, should i count it manually?
or function sntp_get_current_timestamp(); inside time(); only sync current time. (not always call SNTP)
I made some comments on Github. The NTP server is only called every hour.
@@AndreasSpiess So, as your note in github "So there is no need for the function getTimeReducedTraffic()"
Andreas, the link to the DCF77 github page gives lovely 404 error. :( Will you be uploading the code?
Works now. Repo was private. Sorry...
Awsome, thanks. Such a *
I guess the unusual formats @7:57 are due to the week day calculations. The formula is used with Sunday=0.
Check: de.wikipedia.org/wiki/Wochentagsberechnung
Maybe you are right. Back then, one byte was more valuable than today...
In Europe we have next year as last year with summer time... God saved then brain! :)
I am not so sure about that...