ฝัง
- เผยแพร่เมื่อ 1 พ.ค. 2024
- Sometimes we need more memory for our sketches. In this quickie, I show you how you get it without heating up your soldering iron.
Most of the ESP32 modules have 4Mbyte flash memory chips. But if you look at the usable memory size, you only get 1310720 bytes, which is roughly one-quarter of your flash memory. With a blink sketch, this is no problem. It only uses 13% of the available space. But if you use the simple BLE_server example, which is just a few lines long, it occupies already 85% of your flash memory. If you add some other libraries, you quickly get an “out of memory” error.
Partitioning and ESP32 partition tables provide the solution.
Links:
ESP32 boards: s.click.aliexpress.com/e/7xUZti0
Supporting Material and Blog Page: www.sensorsiot.org
Github: www.github.com/sensorsiot
My Patreon Page: / andreasspiess
My Bitcoin address: 19FSmqbBzb5zsYB1d8Bq4KbxVmezToDNTV
If you want to support the channel, please use the links below to start your shopping. No additional charges for you, but I get a commission (of your purchases the next 24 hours) to buy new stuff for the channel
For Banggood bit.ly/2jAQEf4
For AliExpress: bit.ly/2B0yTLL
For ebay.com: ebay.to/2DuYXBp
profile.php?...
/ spiessa
www.instructables.com/member/...
Please do not try to Email me or invite me on LinkedIn. These communication channels are reserved for my primary job
Equipment in my lab: www.sensorsiot.org/my-lab/ - วิทยาศาสตร์และเทคโนโลยี
Cool - I'm using the Adafruit HUZZAH32 and the Partition Scheme menu item doesn't exist but a quick edit of boards.txt put that right and I'm now using Minimal SPIFFS with no problems :) Thank you.
Really neat! Thank you
hello, what kind of edit you do in boards.txt?
@@dhaniyanuar5024 I've just checked the ESP32 Arduino github and it seems that the partition lines are now in boards.txt, as installed.
I've spend probably 4+ hours researching the web, trying to find a simple explanation and resolution to the 1.3MB limit, and your video is exactly what I needed, Thanks!!!😎👍
You are welcome!
Great topic! I was having to fiddle with this a while back too.
I thought it can be useful because I had a few questions about it...
I’m so happy😀😀😀 all my prays answered. Thank you Andreas👍🏴
You are welcome. If it always were so easy ;-)
Awesome information Andreas, very useful indeed!
Thank you!
OMG, this info just saved my life!
:-)
Very useful. I was searching for it everywhere. Thank you.
Glad it was helpful!
Thanks this might come in handy for my home automation project :)
You are welcome!
very usefull, thank you Andreas..
You are welcome!
Back again and about to test this out😀🏴👍
I hope it works now...
Excellent Andreas, as usual.
Thank you!
Great info Andreas. Thanks for the video(s)
You are welcome!
you are the best, in this moment i have this problem thakss!!!!
You are welcome!
Interesting and educative video, thanks for sharing.
You are welcome!
Great quickie thanks.
:-)
Thank you! This is still relevant.
You are welcome!
THANK YOU, YOUR VIDEO WAS VERY HELPFUL. 🤯
Glad you liked it!
Thank you very much! Today you saved my life!! 👊🏻
Glad I could help!
Hello Andreas.
Thank you for another great video.
Why there is no possibility to use No OTA AND minimum SPIFFS at the same time?
exactly - that sounds like a very useful default. It's be good to have someone built and test one and share it ? personally I never use OTA or SPIFFs, but I use multiple libraries in some projects and having most of that 4mb would to use in sketches would be my default for most of the time. weird that whomever made those defaults didn't put that obvious one in, unless it isn't supported for some reason.
If I understand it right, you might try to provide your own partition CSV with no app1 and a minimal SPIFF section. I.e. combine the best parts of no_ota.csv and min_spiffs.csv. (Edit: typo)
There are example partition CSVs at github.com/espressif/arduino-esp32/issues/1476 which answer exactly that. Last one allocates ~3.5MB for a single APP
Thanks that was very helpful
You are welcome!
Thanks a lot for this video!
:-)
Hey Andreas, can you do a video over Esp32 Cam and/or m5stack?
Thank you
I actually stumbled into this one by accident and have been using the minimal SPIFFS configuration as it made more logical sense to me.
Again a great quick video to get this done.
This info was somehow hidden. So I thought I bring it to the attention of everybody.
Very useful information, especially since some of the Wemos D1 boards have 16MB of flash (that is difficult to use). On ESP32, RAM is the barrier I most frequently run into, dreams-first. It encourages creativity.
I think the ESP8266 does not use the same method.
Internally, it's slightly different, but from the IDE the ESP8266 uses a similar process to set target flash partitioning. It can only be set from the Arduino IDE if valid partitioning has been defined in your boards.txt for the currently selected board.
Example linker command line: -Tesp8266.flash.4m1m.ld which yields 3MB program, 1MB SPIFFS.
But from within the Arduino IDE using a Wemos D1 Pro Mini, you'd simply select "3MB Program, 1MB SPIFFS" from the Arduino Tools menu.
You can define custom partitioning, too, but you need to understand the board and linker limitations.
Thanks for sharing 👍😀
You are welcome!
Thank you Andreas, that was very informative I had no idea. Can I assume that the RTOS is responsible for the 13% in the blink sketch? If you added external SD flash memory could that be used for OTA?
OTA is a function programmed by espressif. If you want to use another flash, you probably have to adapt this code.
Many Thanks, that helped me too much
You're welcome!
thank you.. as always helpfull
:-)
greetings from sudan! :)
Never had a comment from you country!
Thanks, very useful !
You are welcome!
Useful video
Thank you!
Thank you!
You are welcome!
Thanks, i wondered was those ota spiffs options were about
Now you know ;-)
Thank you, it is nice to learn something new every day.
Interesting ! As usual.
Thank you!
I love quicky
:-))
Great! Very useful.
Thank you!
usefull info thanks for this
You are welcome!
Hey man, you saved us!!!!!!!!!!!
:-)
Brilliant!
Thank you!
Wow!! That's really neat! I never knew you could change the partition of the memory spaces in these mcu's. I'll have to remember this when I'm working with my custom bms. Can the STM32 also be done this way?
I do not know STM32.
Technically the flash chip is on the WROOM/WROVER module, not on the ESP32. The STM32 seems to have some memory within it at first glance, but I'm not positive. It's formatting would be up to the developers who created the IDE and libraries.
Thanks a lot .
Good information .
Iraq .
You are welcome. Nice to have viewers in your country!
Great video again, Thanks!
I never used OTA and my first thought was to remove app1 and minimize SPIFFS like in the other comments, but then I thought, why not using ota?
I scanned your videos and did not find an esp arduino ota tutorial.
I am sure not only I would be glad if you made one!
Would you be so kind?
Or is it the same as with the ESP8266?
Greetings, Roland (the one with the shortest ESP-video ever ;-) )
There is an example file in the ESP section (ArduinoOTA). I never tried it so far.
I will have to check this next time I'm connected to a board. I have over 500 lines of code with 6-7 libraries loaded and it tells me about 50% used. Maybe I found this setting months ago and just didn't know? I'll edit this comment when I know more.
Technically, OTA doesn't need two equal sized partitions. You could copy the program directly over the original program as it's already loaded on RAM. The reason why there are two partitions is safety. If there's a problem when downloading and the program file is corrupted, a Serial reflash would be needed (which would be difficult and expensive to do in a consumer electronic product). And also, don't forget Secure Boot, used for anti-hacking, important in IoT products so that someone can't upload a malicious code exploiting the OTA update. If an error in the signature is found and the program is on the original partition, it would refuse to boot and, again, a Serial reflash would be needed. And if flash encryption is enabled, it's even more difficult to reflash via Serial. It might even be impossible depending on how you set it up, forcing you to replace the ESP32 chip (even more difficult and expensive). That means an attacker could disable an IoT alarm, for example, by simply uploading an arbitrary, not signed, program! And it would take hours to fix in a best case scenario (the victim has an electronics lab with ESP32s) to days in a worst case scenario (the product has to be sent to a technician and back to the victim)
I think your comment describes why espressif chose to do what they do. And still, it is very easy to brick you ESP yourself with a bad sketch (re.g. blocking loop). Then, OTA does not work and you need to reflash the board :-(
Ture, but at least you can test if it works before releasing the update if you have a consumer electronic product. The other scenarios are more likely to happen.
Great, thanks.
You are welcome!
hi andreas, great video as always, can we have the same results on the esp8266 ?
thank you
I do not know if or how this works on the ESP8266
Does anybody out there know?
It would be really handy if it did. Zero or almost zero Spifs could make OTA viable on an ESP-01, and we've probably all got a draw full of those...
Do not have time to read it right now but this article may have the answer regarding the 8266
tinkerman.cat/eeprom-rotation-for-esp8266-and-esp32/
You are right. At the bottom of the article.
used custom partiton table in platformIO and can confirm that you can make your main partition 3Mb
add *board_build.partitions = partitions_custom.csv* to your *platformio.ini* and your partitions_custom.csv file to the root of the project.
Thank you for your tip!
thank you
You're welcome
very interesting subject. thanks for sharing your knowledge. would be possible to do the same using micropython instead of Arduino IDE?
I do not know how microPython uses the memory. But I think they already use it completely.
This is kind of off the topic for memory but i am having a hard time trying to set a static IP for my ESP32's. have you been able to get this to work?
No. I always use dynamic IP.
This didn't really show the functions to fetch the partition sizes...is that available? I'm curious, because I have a v2 Heltec Lora, which has 8MB QSPI flash and I wanted to just check the allocations. I would love to find this function. Also, I might mention I'm running in VSC with platformIO and not sure if you can set those default partitions there or how. As usual, you, sir, are a master and make very excellent/quick videos that answer some very complex questions! Thank you!
...and a little poking around on espressif's site: docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/spi_flash.html#partition-table-api
Usually you define the partition scheme in the IDE before you compile. So I do not see a big need to do that after deployment. You find the partition schemes if you chose the ESP32dev board. You should also find these constants in PIO, but you have to google.
Everybody loves a quickie from time to time. Is OTA standardized somehow for ESP32? Or do I need to write my own code to be able to use it? Thank you!
I think it is part of the IDE. Only a few lines of code.
@@AndreasSpiess thank you! I'll give it a try!
Or... do you think that OTA would fit one of your future viseos?
Is it possible to allocate the total available SPIFFS for the sketch only, I don’t require the OTA or the capability to store other files I want all the memory available for the sketch only. Please help.
Usually you can choose the chose the partitioning in the Arduino IDE (Tools-->partition scheme)
@@AndreasSpiess i know but, i want to allocate all the memory for the sketch only with no SPIFFS or no OTA,default options has either combinations. Please help me i have been searching for a long time but didn’t manage to find a reasonable answer.
Very useful to know, yet I haven't encountered this problem yet.
You never know when you will encounter it ;-)
Can you explain how to make a memory map of current allocations and usage? thanks!
I do not know how to do that. I only know how to create a new one.
Great tip, aren't there esp32s with 16mb available?
Maybe. I never had the need so far.
sort of. analoglamb sells some custom ESP32's with more storage.
nice video! Can you give advice using ESP32 in sleep mode? Battery keeps dying..
I did it once for my board tests. I only used the example file. But I did not try if something is broken now. That would be an issue for Github...
There are a number of Deep Sleep examples built into the esp32 library.
I see that using the esp_sleep_enable_ext0_wakeup() method is the way to go. Is there a possibility to use as a wake-up event an incoming I2C message?
I do not think so. I2C needs a running processor.
I would concur. Proper method would likely be waking up periodically and polling for new data via i2c. Best method is to devote a pin to waking up via an interrupt, which is supported.
this works on Esp8266 aswell right?..i need to upgrade to a Esp32
I do not think so. At least, I do not know.
Back again
4:30 "if true" 😂 dude you are compiling LIFE
I do not understand...
@@AndreasSpiess I thought that the only thing missing from the expression at 4:30 was a semicolon at the end :D You of course could have done this deliberately though. Have a good day sir. Guata Morga👍
depends on the language you use ;-)
@@AndreasSpiess indeed :D
Hi Andreas, great video, I was wondering if you can assist me, I'm using doit esp32 dev kit v1, and I can't see the partition scheme option under tools... Is there somehitng else that I should be doing?
Just use the "ESP32 DEV" module" board instead, or "transplant" the lines in boards.txt to your board (As shown in the video.
Line transplantation worked for me flawlessly on that board.
In Arduino IDe is it possible to display the size of SPIFFS ?
Thanks :)
Marc.
Usually you can display the partition scheme where you define the SPIFFS size
There are API functions that will return used and available bytes. Check out the SPIFF library, SPIFFS.h file for the declaration.
Hi. I was trying to get the partition plugin for arduinoIDE (francis94c/ESP32Partitions) working but did not succeed. I noticed your conversation with this plugin writer and is seems as if you got it to work. I wonder if you would consider making a video as how you manged to install it and getting it to work. thanks....
I use the ESP32DEVboard in the Arduino IDE where I can chose the partition i need.
It's this still true in later versions of the ide? I cannot find that partition menu option. At least not in versions 1.8.8 and later
It is still here on my 1.8.9 version. But only with ESP32 boards. Try once "ESP32 dev module" board
@@AndreasSpiess appreciate the reply. Sure enough changing the board to ESP dev module had updated the menu and i see the function. thanks!
very interesting :D
:-)
Oh my god, thank you so much ! You are a very-pretty-you-tuber yourself
I was wondering where are my lost 3 MB for a very long time and always felt a bit ... cheated
Until you came along :) so easy, and so many new exciting possibilities
You are welcome!
Don't forget that the esp32 runs an rtos in the background so that will also take up a lot of memory.
But it also has a lot of memory ;-)
Please explain how to setup the partition tables for aerduino IDE 2
I do not understand "aerduino IDE 2"
@@AndreasSpiess sorry my spelling. I ment the new Arduino 2 IDE. I was asking how to get the partions menu setup under tools menu.
@@royhopkins974 It is at the same place as in the old IDE. But you have to select ESP32 DEV module. Otherwise, it might not be displayed.
So could you do a "Min SPIFF No OTA" custom partition scheme and get practically all of the memory available for one app?
Yes: github.com/espressif/arduino-esp32/issues/1476
Is it the same for the AIThinker esp-CAM?
I think so.
Hello, what if ... I do not know: What if you manage OTA over an additional sd-card memory. It must not be fast. Is this possible?
I do not know. The OTA code is maintained by espressif and I would not want to change it...
Oh, I thought the code is a library, easy to change so that was the missing info I had.
Maybe technically it is in a library. I am not sure. But it is done by espressif
ok. Good Night. Its late in Europe!
is it possible (and/or advisable) to get rid of spiffs entirely? Directly flash?
It depends on your needs. You can put the variables in your sketch (flash memory), use EEPROM (also if flash memory), or SPIFFS (also in flash memory). All with advantages and disadvantages.
Yes, just make do not include a spiffs partition in your partitions.csv file. Your program goes in the "factory" partiition. "NVS" can be used by you to store variables, structs etc but is also used by the system so keep that in there.
I can't find a partition scheme in the tools bar, I am using Arduino IDE 1.8.12.
is there any way to solve this?
It depends on which board you selected. Not all show this option.
This option only works with the "ESP32 dev module", currently i'm working with the "DOIT ESP32 DEVKIT V1", if i upload a sketch using the first board, do i keep the partition definition if i upload a second sketch but using my actual board?, if not, is there a way to add such optiion to any board model?.
Yes, you have to copy the lines of the boards.txt file to your board definition. But you also can use the ESP32 dev module board. I always use that one.
I see, thanks for the info!
tussi great ho G !
???
@@AndreasSpiess it means U are legend ! did a great job (comment was in Punjabi)
Thank you!
I didn´t understand what is OTA in ESP32. What is it useful for? Thanks
You can update your firmware over WiFi. Very useful if you mounted your ESP in an inaccessible place.
If some esp32 has 4mb ram, does it help?
All ESP32 have at least 4MB flash
I am not seeing Partition Scheme in the Tools menu of the IDE, tried on 1.8.5 and 1.8.6. Has anyone got any ideas please.. Thank you
Not all boards definitions support partitioning. The ESP32 dev Module definition supports it.
You can add these lines to your board:
esp32.menu.PartitionScheme.default=Default
esp32.menu.PartitionScheme.default.build.partitions=default
esp32.menu.PartitionScheme.minimal=Minimal (2MB FLASH)
esp32.menu.PartitionScheme.minimal.build.partitions=minimal
esp32.menu.PartitionScheme.no_ota=No OTA (Large APP)
esp32.menu.PartitionScheme.no_ota.build.partitions=no_ota
esp32.menu.PartitionScheme.no_ota.upload.maximum_size=2097152
esp32.menu.PartitionScheme.min_spiffs=Minimal SPIFFS (Large APPS with OTA)
esp32.menu.PartitionScheme.min_spiffs.build.partitions=min_spiffs
esp32.menu.PartitionScheme.min_spiffs.upload.maximum_size=1966080
in boards.txt
please replace the esp32. with the text of your board.
Andreas Spiess I worked on it last night and solved why it wasn’t working. I was working off the old esp32 core. Once I deleted it and added esp32 from within the Arduino IDE I had the options presented to me😀 many thanks for the code. I will store it somewhere safe. My BLE sketch has complied and uploaded into the new partition and all looking great 😀😀🏴
Hi, just asking where we can get access to the Partition Scheme Tool. I don't seem to have access to all the tools :(
Maybe you chose a different board in the IDE definition. Not all boards have all definitions implemented. I select the "plain vanilla" dev board.
@@AndreasSpiess Ah ok. I was using the 'DOIT ESP32 DEVKIT V1' board and thought you needed to download the tools from somewhere. Thanks for the help :)
👍👏
:-)
They also sell ESP32 with 8/16/32Mb flash size for like $1 or $2 more
Do you have a link?
@@AndreasSpiess
www.electrodragon.com/product/wroom-32/
www.analoglamb.com/product/esp32-wrover/
Thanks! Were you able to use the flash in the Arduino IDE?
Yes. They work out of the box
I didn't need SPIFFS but still needed OTA , i tried this configuration and it works with HTTP OTA + BLE + MQTT. please do comment if you think there is an issue with this.
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x5000,
otadata, data, ota, 0xe000, 0x2000,
app0, app, ota_0, , 0x1f7f00,
app1, app, ota_1, , 0x1f7f00,
eeprom, data, 0x99, , 0x200,
spiffs, data, spiffs, , 0x0,
i tried to maximize app0 and app1 for this one. also i left the offset blank because the esp_tool can handle offset alignment as per the document.
I cannot do remote debugging :-(
@@AndreasSpiess this is interesting.
btw, i updated mine to elimate eeprom and maximised app1, app0.
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x5000,
otadata, data, ota, 0xe000, 0x2000,
app0, app, ota_0, 0x10000, 0x1f0000,
app1, app, ota_1, 0x200000, 0x1f0000,
eeprom, data, 0x99, 0x3f0000, 0x0,
spiffs, data, spiffs, 0x3f0000, 0xf000,
unfortunately my requirement includes support for Wifi OTA, WIFI Mqtt, and BLE Server for setting up node configuration via smart phone.
I've been able to successfully merge all these 3 plus a variety of peripheral support on the device but somehow i needed to do a software restart every time the user needs to change mode (wifi mqtt as client -> ble server -> wifi ota as server) . it turned out, managing spiffs is easier than using eeprom with arduinojson.
i never did remote debugging before and i don't think it will be needed in the product im involved with.
this particular video explanation you posted has been instrumental in bootstrapping my knowledge in esp32 partitioning. thank you very much!
this is strange. Micropython uses ~500 kb out of 4 MB. The only memory problem micropython gives is RAM memory shortage and only on esp8266. I am surprised that Arduino eats so much disk space.
Micropython uses a different concept. It stores its "sketch" in RAM. This is why the ESP8266 is not used for big micropython projects and they use ESP32 with PSRAM
i'm probably missing something here, but this caught my attention just now and it feels weird.
the blink sketch uses a couple standard arduino library functions (3 or 5, depending on how you count setup() and loop()) and - compiled - is around 170 KiB.
a hello world program in C on my computer (linux, gcc, amd64) compiles to a 7.8 KiB binary.
what part am i missing here that explains why the microcontroller binary is a solid order of magnitude larger than the one on amd64? in particular, if you consider that the blink sketch essentially only sets a few registers. it just doesn't add up for me.
Your C program will dynamically load the standard C library, which is big. Since it's dynamically loaded the file of the executable remains small.
ah, thanks - yes, that was the part that i was missing in my brainfart. now it makes sense.
cheers!
Aside from the C library the Arduino library itself forces it to link to many stuff that you don't see or make use. Arduino inherits from it's Wiring past the fact that it is not well designed for speed or small footprint, but portability except in MCUs that have low storage space, obviously.
For actual Arduino boards with AVR microcontrollers you can use Atmel Studio and don't use the Arduino library (or IDE) and that will give you a much smaller foot print, and also teach you more closely how your MCU works.
thanks for the detail! i guess the the arduino ide does just fine for most purposes, but this is definitely something to keep in mind for that one occasion where you run of memory :)
thank you so much from morocco
You are welcome! Beautiful country. I was once on a desert race with my bike and once in Marrakesh.
You are welcome
I'm using PlatformIO, a much better IDE than Arduino IDE. Easy to change the partitioning both for ESP8266 and for ESP32. An example here for ESP8266 with almost 2MB for app0 and app1, github.com/espressif/arduino-esp32/issues/1476#issuecomment-395348594
Thank you for the link!
Hello Andreas,
I admire your work and channel; and thanks for inspiring.
I have a suggestion, I co-built an arduino plugin with a friend to automate memory partition for esp32
See latest Esp32 Partition manager:
github.com/francis94c/ESP32Partitions
Other examples of usage is in:
github.com/espressif/arduino-esp32/tree/master/libraries/EEPROM/examples/eeprom_class
Best regards,
Elochukwu Ifediora (fedy0)
Interesting concept. However, it did not run for me. I added my point to the existing issue.
I tried this tool too hoping it would solve all of my problems but no tool in the menu - tried various versions of the arduino IDE and still couldn’t get the gui tool to show 😢 thanks for your effort nonetheless 🙏
great video. plz see "Electronics Plus" app. Where we added your video in learning section.
Thank you. I hope you mentioned my channel ;-)
@@AndreasSpiess yes. we directly give your channel. plz download the "Electronics Plus" app from play store
Just because I can, FIRST!!!!!!
Give me soildering iron please very need no money and studend
Soldering irons can be found for $2-5 on eBay. Don't beg someone providing you with free information for hardware as well.