esp8266 firmware and filesystem update from SD card – 1

by · Published · Updated

Spread the love
           

In this series of article about firmware and how to update It, I’d like to add a series of alternative methods of update very useful for me.

esp8266 firmware and filesystem update from SD card
esp8266 firmware and filesystem update from SD card

In this article, we will learn how to use external storage like the SD card to update firmware and filesystem.

Update firmware from a storage with Update class

Another interesting system to update firmware is to use an SD card or local filesystem. To do that, we can use the Update class.

A method, in particular, can manage update via stream

Update.begin(firmwareSizeInBytes);
Update.writeStream(streamVar);
Update.end();

Use SD card for update

A widely used system to update firmware is to add a file inside external storage like removable SD and at boot of the device update with It.

In this case, we are going to follow these steps:

  1. Insert a firmware named firmware.bin in the SD;
  2. At boot check if firmware.bin exist;
  3. If exist start the stream/update of the file;
  4. At the end rename firmware.bin in firmware.bak, so we stop the loop of update;
  5. Reset and reboot the microcontroller.

Wiring SD adapter

First, we must wire the SD, and you can follow this tutorial “How to use SD card with esp8266 and Arduino“.

WeMos D1 mini esp8266 SD adapter wiring
WeMos D1 mini esp8266 SD adapter wiring

Firmware update sketch via SD card

Sketch OTA update File system EEPROM WiFi config

The sketch is quite simple, and I will explain It in detail.

/*
 * Update firmware with external SD
 * Check firmware.bin, if present start update then rename in firmware.bak
 *
 * Renzo Mischianti <www.mischianti.org>
 *
 * https://mischianti.org/
*/

// include the SD library:
#include <SPI.h>
#include <SD.h>

// WeMos D1 esp8266: D8 as standard
const int chipSelect = SS;

void progressCallBack(size_t currSize, size_t totalSize) {
	  Serial.printf("CALLBACK:  Update process at %d of %d bytes...\n", currSize, totalSize);
}

#define FIRMWARE_VERSION 0.1

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }


  Serial.print("\nInitializing SD card...");

  // we'll use the initialization code from the utility libraries
  // since we're just testing if the card is working!
  if (!SD.begin(SS)) {
    Serial.println("initialization failed. Things to check:");
    Serial.println("* is a card inserted?");
    Serial.println("* is your wiring correct?");
    Serial.println("* did you change the chipSelect pin to match your shield or module?");
    while (1);
  } else {
    Serial.println("Wiring is correct and a card is present.");
  }

  FSInfo fs_info;
  SDFS.info(fs_info);

  Serial.print("Total bytes: ");
  Serial.println(fs_info.totalBytes);

  Serial.print("Used bytes: ");
  Serial.println(fs_info.usedBytes);

  Serial.print(F("\nCurrent firmware version: "));
  Serial.println(FIRMWARE_VERSION);

  Serial.print(F("\nSearch for firmware.."));
  File firmware =  SD.open("/firmware.bin");
  if (firmware) {
	  Serial.println(F("found!"));
	  Serial.println(F("Try to update!"));

	  Update.onProgress(progressCallBack);

	  Update.begin(firmware.size(), U_FLASH);
	  Update.writeStream(firmware);
	  if (Update.end()){
		  Serial.println(F("Update finished!"));
	  }else{
		  Serial.println(F("Update error!"));
		  Serial.println(Update.getError());
	  }

	  firmware.close();

	  if (SD.rename("/firmware.bin", "/firmware.bak")){
		  Serial.println(F("Firmware rename succesfully!"));
	  }else{
		  Serial.println(F("Firmware rename error!"));
	  }
	  delay(2000);

	  ESP.reset();
  }else{
	  Serial.println(F("not found!"));
  }


}

void loop(void) {
}
WeMos D1 mini esp8266 SD adapter wiring photo
WeMos D1 mini esp8266 SD adapter wiring photo

First, we try to open firmware.bin

  File firmware =  SD.open("/firmware.bin");
  if (firmware) {

If the file exists, start the stream of the file to the OTA partition 

	  Update.begin(firmware.size(), U_FLASH);
	  Update.writeStream(firmware);
	  if (Update.end()){

And follow the progress with the relative callback

	  Update.onProgress(progressCallBack);

In the end, I rename firmware.bin to prevent another update at reboot.

	  if (SD.rename("/firmware.bin", "/firmware.bak")){
		  Serial.println(F("Firmware rename succesfully!"));
	  }else{
		  Serial.println(F("Firmware rename error!"));
	  }

Finally, I restart the esp to allow the copy of the firmware from the OTA partition to the flash.

Execution of the test

For the test, we will remove SD and upload the sketch with this version of the firmware.

#define FIRMWARE_VERSION 0.1

Then change the version to 0.2

#define FIRMWARE_VERSION 0.2

And generate compiled binary refers to the article “esp8266: flash compiled and signed firmware“

esp8266 esp32 Arduino Export compiled binary IDE menu
esp8266 esp32 Arduino Export compiled binary IDE menu

Now you can get the bin file from the sketch folder (Sketch -> Show Sketch Folder), copy it to the SD, and rename it as firmware.bin.

When you reinsert the SD and click the reset, you can get this serial output.

Initializing SD card...Wiring is correct and a card is present.
Total bytes: 3967680512
Used bytes: 458752

Current firmware version: 0.10

Search for firmware..found!
Try to update!
CALLBACK:  Update process at 0 of 309424 bytes...
CALLBACK:  Update process at 4096 of 309424 bytes...
CALLBACK:  Update process at 8192 of 309424 bytes...
CALLBACK:  Update process at 12288 of 309424 bytes...
CALLBACK:  Update process at 16384 of 309424 bytes...
CALLBACK:  Update process at 20480 of 309424 bytes...
CALLBACK:  Update process at 24576 of 309424 bytes...
CALLBACK:  Update process at 28672 of 309424 bytes...
CALLBACK:  Update process at 32768 of 309424 bytes...
CALLBACK:  Update process at 36864 of 309424 bytes...

[...]

CALLBACK:  Update process at 270336 of 309424 bytes...
CALLBACK:  Update process at 274432 of 309424 bytes...
CALLBACK:  Update process at 278528 of 309424 bytes...
CALLBACK:  Update process at 282624 of 309424 bytes...
CALLBACK:  Update process at 286720 of 309424 bytes...
CALLBACK:  Update process at 290816 of 309424 bytes...
CALLBACK:  Update process at 294912 of 309424 bytes...
CALLBACK:  Update process at 299008 of 309424 bytes...
CALLBACK:  Update process at 303104 of 309424 bytes...
CALLBACK:  Update process at 307200 of 309424 bytes...
CALLBACK:  Update process at 309424 of 309424 bytes...
CALLBACK:  Update process at 309424 of 309424 bytes...
Update finished!
Firmware rename succesfully!

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v0005b4c0
@cp:B0
ld

Initializing SD card...Wiring is correct and a card is present.
Total bytes: 3967680512
Used bytes: 458752

Current firmware version: 0.20

Search for firmware..not found!

Update filesystem from a storage with Update class

It’s possible to specify what you want to update FLASH (firmware) or FS (filesystem).

Now we will show how to manage the filesystem, but we go fast because the changes were minimal.

Generate the file with the binary file system

To manage the filesystem, we can use the same way, but first, you must read one of the following guides:

WeMos D1 mini (esp8266), integrated SPIFFS Filesystem (deprecated)

WeMos D1 mini (esp8266), integrated LittleFS Filesystem (recommended)

We will add the data directory to the sketch folder and use the plugin to upload it.

esp8266 LittleFS plugin menu on ArduinoIDE
esp8266 LittleFS plugin menu on ArduinoIDE

In the console, we find the right command to generate this file.

[LittleFS] data    : D:\tmp\testBlink\Blink\data
[LittleFS] size    : 2024
[LittleFS] page    : 256
[LittleFS] block   : 8192
/bombo.png
[LittleFS] upload  : C:\Users\renzo\AppData\Local\Temp\arduino_build_49036/Blink.mklittlefs.bin
[LittleFS] address : 0x200000
[LittleFS] reset   : --before default_reset --after hard_reset
[LittleFS] port    : COM17
[LittleFS] speed   : 921600
[LittleFS] python   : C:\Users\renzo\AppData\Local\Arduino15\packages\esp8266\tools\python3\3.7.2-post1\python3.exe
[LittleFS] uploader : C:\Users\renzo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.7.4\tools\upload.py

esptool.py v2.8
Serial port COM17
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 50:02:91:74:fe:11
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 2072576 bytes to 48389...
Wrote 2072576 bytes (48389 compressed) at 0x00200000 in 4.9 seconds (effective 3416.7 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Now we can grab a lot of information; first, the file

[LittleFS] upload  : C:\Users\renzo\AppData\Local\Temp\arduino_build_49036/Blink.mklittlefs.bin

Sketch to update filesytem with a file in the SD card

I reuse the previous sketch, and I’m going to add a section to manage the firmware update.

/*
 * Update firmware with external SD
 * Check firmware.bin, if present start update then rename in firmware.bak
 *
 * Update filesystem with external SD
 * Check filesystem.bin, if present start update then rename in filesystem.bak
 *
 * Renzo Mischianti <www.mischianti.org>
 *
 * https://mischianti.org/
*/

// include the SD library:
#include <SPI.h>
#include <SD.h>

// WeMos D1 esp8266: D8 as standard
const int chipSelect = SS;

void progressCallBack(size_t currSize, size_t totalSize) {
	  Serial.printf("CALLBACK:  Update process at %d of %d bytes...\n", currSize, totalSize);
}

#define FIRMWARE_VERSION 0.1

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }


  Serial.print("\nInitializing SD card...");

  // we'll use the initialization code from the utility libraries
  // since we're just testing if the card is working!
  if (!SD.begin(SS)) {
    Serial.println("initialization failed. Things to check:");
    Serial.println("* is a card inserted?");
    Serial.println("* is your wiring correct?");
    Serial.println("* did you change the chipSelect pin to match your shield or module?");
    while (1);
  } else {
    Serial.println("Wiring is correct and a card is present.");
  }

  FSInfo fs_info;
  SDFS.info(fs_info);

  Serial.print("Total bytes: ");
  Serial.println(fs_info.totalBytes);

  Serial.print("Used bytes: ");
  Serial.println(fs_info.usedBytes);

  Serial.print(F("\nCurrent firmware version: "));
  Serial.println(FIRMWARE_VERSION);

  Serial.print(F("\nSearch for firmware.."));
  File firmware =  SD.open("/firmware.bin");
  if (firmware) {
	  Serial.println(F("found!"));
	  Serial.println(F("Try to update!"));

	  Update.onProgress(progressCallBack);

	  Update.begin(firmware.size(), U_FLASH);
	  Update.writeStream(firmware);
	  if (Update.end()){
		  Serial.println(F("Update finished!"));
	  }else{
		  Serial.println(F("Update error!"));
		  Serial.println(Update.getError());
	  }

	  firmware.close();

	  if (SD.rename("/firmware.bin", "/firmware.bak")){
		  Serial.println(F("Firmware rename succesfully!"));
	  }else{
		  Serial.println(F("Firmware rename error!"));
	  }
	  delay(2000);

	  ESP.reset();
  }else{
	  Serial.println(F("not found!"));
  }

  Serial.print(F("\nSearch for filesystem update.."));
  File filesystem =  SD.open("/filesystem.bin");
  if (filesystem) {
	  Serial.println(F("found!"));
	  Serial.println(F("Try to update!"));

	  Update.onProgress(progressCallBack);

	  Update.begin(filesystem.size(), U_FS);
	  Update.writeStream(filesystem);
	  if (Update.end()){
		  Serial.println(F("Update finished!"));
	  }else{
		  Serial.println(F("Update error!"));
		  Serial.println(Update.getError());
	  }

	  filesystem.close();

	  if (SD.rename("/filesystem.bin", "/filesystem.bak")){
		  Serial.println(F("Filesystem rename succesfully!"));
	  }else{
		  Serial.println(F("Filesystem rename error!"));
	  }
	  delay(2000);

	  ESP.reset();
  }else{
	  Serial.println(F("not found!"));
  }


}

void loop(void) {
}

The relevant information that you can find here if this

	  Update.begin(filesystem.size(), U_FS);

I changed U_FLASH in U_FS; this internally set the correct address of the flash section and started updating from there.

Test the sketch

We create a version.txt File on data folder of the sketch and I write 0.1 inside, and I use that as the version of the FileSystem, then upload the data folder to the filesystem.

Initializing SD card...Wiring is correct and a card is present.
Total bytes: 3967680512
Used bytes: 655360

Current firmware version: 0.10

Search for firmware..not found!
Firmware version 0.10
Inizializing FS...done.
FileSystem version 0.1

Search for filesystem update..not found!

Now we are going to modify the file version.txt with the version 0.2 then upload to the device, then

Change version to 0.2 in version.txt, regenerate without upload, and copy the file ArduinoOTAesp8266_sd_fs_update.mklittlefs.bin to the SD and rename to filesystem.bin.

Now insert SD and check the serial output.

Initializing SD card...Wiring is correct and a card is present.
Total bytes: 3967680512
Used bytes: 2228224

Current firmware version: 0.10

Search for firmware..not found!
Firmware version 0.10
Inizializing FS...done. 
FileSystem version 0.1

Search for filesystem update..found!
Try to update!
CALLBACK:  Update process at 0 of 2072576 bytes...
CALLBACK:  Update process at 4096 of 2072576 bytes...
CALLBACK:  Update process at 8192 of 2072576 bytes...
CALLBACK:  Update process at 12288 of 2072576 bytes...
CALLBACK:  Update process at 16384 of 2072576 bytes...
CALLBACK:  Update process at 20480 of 2072576 bytes...
CALLBACK:  Update process at 24576 of 2072576 bytes...
CALLBACK:  Update process at 28672 of 2072576 bytes...
CALLBACK:  Update process at 32768 of 2072576 bytes...
CALLBACK:  Update process at 36864 of 2072576 bytes...
CALLBACK:  Update process at 40960 of 2072576 bytes...
CALLBACK:  Update process at 45056 of 2072576 bytes...
CALLBACK:  Update process at 49152 of 2072576 bytes...
CALLBACK:  Update process at 53248 of 2072576 bytes...
CALLBACK:  Update process at 57344 of 2072576 bytes...
CALLBACK:  Update process at 61440 of 2072576 bytes...
CALLBACK:  Update process at 65536 of 2072576 bytes...
CALLBACK:  Update process at 69632 of 2072576 bytes...
CALLBACK:  Update process at 73728 of 2072576 bytes...
CALLBACK:  Update process at 77824 of 2072576 bytes...

[...]

CALLBACK:  Update process at 2052096 of 2072576 bytes...
CALLBACK:  Update process at 2056192 of 2072576 bytes...
CALLBACK:  Update process at 2060288 of 2072576 bytes...
CALLBACK:  Update process at 2064384 of 2072576 bytes...
CALLBACK:  Update process at 2068480 of 2072576 bytes...
CALLBACK:  Update process at 2072576 of 2072576 bytes...
CALLBACK:  Update process at 2072576 of 2072576 bytes...
Update finished!
Filesystem rename succesfully!

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v00051e80
~ld

Initializing SD card...Wiring is correct and a card is present.
Total bytes: 3967680512
Used bytes: 2228224

Current firmware version: 0.10

Search for firmware..not found!
Firmware version 0.10
Inizializing FS...done. 
FileSystem version 0.2

Search for filesystem update..not found!

First time find the filesystem.bin then change the name and next time do nothing.

Thanks

  1. Firmware management
    1. esp8266: flash firmware binary (.bin) compiled and signed
    2. esp8266: flash firmware and filesystem binary (.bin) compiled with GUI tools
  2. OTA update with Arduino IDE
    1. esp8266 OTA update with Arduino IDE: filesystem, signed and password
  3. OTA update with Web Browser
    1. esp8266 OTA update with Web Browser: firmware, filesystem and authentication
    2. esp8266 OTA update with Web Browser: sign the firmware and HTTPS (SSL/TLS)
    3. esp8266 OTA update with Web Browser: custom web interface
  4. Self OTA uptate from HTTP server
    1. esp8266 self OTA update firmware from server
    2. esp8266 self OTA update firmware from server with version check
    3. esp8266 self OTA update in HTTPS (SSL/TLS) with trusted self signed certificate
  5. Non standard Firmware update
    1. esp8266 firmware and filesystem update from SD card
    1. esp8266 firmware and filesystem update with FTP client

Support forum

Spread the love
           

Tags:

2 Responses

  1. Vincent says:
    15 February 2023 at 07:28

    I’m sorry if I’m missing the point here. But as I understand it you are uploading a program to your ESP32 and then getting that to pull the firmware from an SD card. Wouldn’t it just be easier rather than do this just upload the ESP32 with the firmware.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Terms and Conditions - Privacy Policy latest - Privacy Policy with company name - Privacy Policy