Esp32 wifi/co-processor hat

im looking to assemble a team for the first pokitto hat.
to extend to most functionality using the esp32 enabling pokitto to go wireless

  • bluetooth offering local wireless multiplayer to another pokitto or compatible device
  • wifi to connect to the internet and open up possibility for online multiplayer
  • a powerfull programable dual core chip to render and compute primitive 3D in realtime

the hardware is the easy part, building a solid api around it is the challenge
one problem is how to program the esp32 for specific tasks, i see 2 solution:

  1. let the pokitto act as a programmer and upload a hex file from sd card
  2. pre install a interpreter (lua or other) and send string data (feels a bit like glsl in a way)

please tell me if your interested in pursuing this project.


I can’t join the team because I’ve got other projects going on and I don’t own an ESP32, but I wouldn’t mind chipping in some advice here and there, especially where advanced C++ techniques and good code style are concerned.

A ‘few’ thoughts to start off:

In terms of technique, I don’t know the ins and outs of the ESP32 but I’d recommend treating it like a GPU or other kind of support chip. Ideally commands should be in some form of bytecode, strings chomp up too much memory and processing power and are non-trivial to interpret.

There should be some means of sending commands (again, preferably byte-based, a bit like SPI screens have) from the Pokitto’s CPU to the ESP32.
These should probably be high-level to abstract away some of the details, e.g. no “set address X to value Y” commands, instead have stuff like “Open TCP connection to IPv4 address X on port Y” and “Close connection Z” - essentially closer to function calls than to machine code. A bit like how DOS abused interrupt vectors to allow programs to do system calls to handle stuff like console printing.

To get the most out of it you might want to upload ‘programs’ to the ESP32 so that there’s less overall slowdown associated with using the ESP32. I.e. instead of continually sending commands every few instructions, upload whole ‘programs’ so the ESP32 can handle some of the processing instead (essentially equivalent to GPU Shaders).

The programs could either be in bytecode or actual ESP32 machine code. Bytecode makes it easier to swap the ESP32 for some other device and possibly less prone to malicious use, but machine code would run faster.

I could be completely wrong or completely crazy.
(I’m definitely at least partly crazy.)

1 Like

well if your ever interested i can probably get you a devboard.

i like the idea of using byte commands though most definitely will be sending a string of lua or compiled lua code up
seems easy enugh for the pokitto to read from sd card and transmit, or other language but though it has to be powerfull and compact, im not sure myself if it can run bytecode on its, i mean it can but swapping it out during runtime and how to get the code uploaded sounds like a mess

also theres a some options in connections both devices have i2c, spi and uart
esp also seems to have a host controler for a bunch of memory storage (SD/SDIO/CE-ATA/MMC/eMMC) maybe it should have its own sd card reader?

im not sure if we can pull the sd card or display from the PEX

It’s not that I’m not interested, it’s just that I’ve got other projects to be doing and I don’t want to try to take on too many.

If you do go down the Lua route, send compiled Lua, not the source.

If you use compiler Lua (i.e. Lua bytecode) then only the Lua environment itself is needed.
If you allow Lua sourcecode you have to include the compiler and that’s not going to be small, even if Lua is quite a simple language.

Also, unless you want the player to be able to program the chip, you don’t need source code, you can precompile the code and load the compiled code onto the Pokitto SD card or even embed it in memory as an array of uint32_t (all Lua bytecode instructions are 32 bits wide).

You’d have to track the lifetime of the code and know when it’s no longer needed.
C++ has two good mechanisms that could probably help - destructors and smart pointers.

The biggest bottleneck is most likely going to be the communication between the ESP and the Pokitto, so that’s what the ‘program upload’ approach minimises.

Think of it like this…
Person A needs to get some work done, but they also need lunch, so so person A asks person B to make lunch.
Person B doesn’t know how to cook.
Person A has two choices:

  1. Get up every 2 minutes to person B what to do next
  2. Give person B a recipe and tell them to follow the recipe

By uploading a program to the ESP, the Pokitto would be giving the ESP a ‘recipe’.

Ultimately though, the best appraoch will depend on the scope of your library.
Thus, I think rather than trying to decide how to do it you should start by deciding what things you want the library to be able to do.

You don’t have to make the hat do all the things at once,
you could have one library for wireless bluetooth and one for wifi,
and maybe have only one library used at once.

That would decrease the complexity of the code,
but it would also increase the cost of the hat.

Neither choice is wrong, it depends on which is more important.

I think both would need other code in place, and again speed might be an issue.

If you’re interested in Lua by the way, the one of the best things you can read about it is the No-Frills introduction to Lua 5.1 instructions/bytecode, which can be found here.

lua sounds like a good option to me uploading the at startup or even ability to swap it out wen the game needs it
also i dont thing diferent library’s would be needed then wen i just extend lua functions with bluethooth and wifi comands so if in case you need both wifi (for pull the world clock or somthing) and bluethooth to do a local multiplayer it be posible

the only concern i have is that lua is large (32bits per instuction i feel is large)
i have seen nes emulators on the esp and even doom, i dont know wich is the best solution

Definitely. It’s better to have a few loading screens than to have a stuttery game.

Other bytecode would be capable of that depending on how it’s designed. In Lua’s case I think it swaps out functions that aren’t in use.

That would be possible to do with custom-made bytecode anyway.
Essentially all bytecode involves is reading some bytes and deciding what to do based on their value, which isn’t much more complicated than reading a file format.

To do that with Lua you’ll need to know how to use the C API to register a C+±side function. That’s explained in section 26.1 of the official Lua C API tutorial.

The main reasons I suggested having two separate libraries are separation of concerns(i.e. if someone only needs the bluetooth they can leave out the wifi stuff entirely) and to keep the libraries simpler (i.e. so you can focus on just one thing at a time).

It depends on the overhead of the system required to handle the bytecode. Lua is considered to be quite lightweight in comparison to other scripting languages of a similar nature and it’s actually designed to be lightweight.

4 bytes is very little when you consider that the ESP32 has 524288 bytes of SRAM. And the importance isn’t just in the size of the instruction, it’s in the size of the ‘programs’ (which in terms of source code may only be a few lines, and the bytecode is almost always smaller than the source).

The NES is actually a fairly complex system to emulate so the ESP is suitably powerful. Probably not as complex as the GBA but more complex than a Chip8.

(I tried to write a NES emulator once, a long time ago. I got most of the M6502 system working and then fell down at the PPU. The PPU is evil.)

1 Like

oh also theres this thing

i would argue against that, with having them as c api function already there on device since the flash would not be reprogramed all these wont leave any overhead there just there to be called if needed its just an extension the the io api, im not to fermiliar with the wifi system but im asuming i need a couple functions like “scan for hotsport/router, conect to router, send recive packets”
i might be wrong with that looking at simplifieng the wifi and bluetooth api parts as more of read write functions.

i do am wondering about other stuff like could add the 3D rasterizer in c for faster code
also wondering if i can send a frame to the pokitto over spi or is it beter to send 2D triangle data

Every function has overhead: the memory cost of the function’s existance.

Not needing reprogramming is a valid point though.

(Though it might be possible for the Pokitto to reprogram the ESP.)

It depends on how much functionality you want.

There are many different layers and protocols to the internet.
If you just want to support simple UDP and/or TCP then it won’t be much, you’ll need to handle choosing a port and IP address and maybe some DNS lookup, but if you wanted HTTP on top of that things start to add up.
(The ESP will probably also need to handle ARP and one of either DHCP, BOOTP or reverse ARP.)

A lot of that can be abstracted away easily enough, but when something goes wrong the diagnostic messages might get complicated.

Probably best to worry about the other stuff first.

It’s certainly possible to send a frame over SPI but I have no idea how long that would take and it could be a bottleneck. Most likely the PEX doesn’t connect to the screen so the Pokitto’s CPU would have to redirect the data coming in from the PEX to either a buffer or the screen. Depending on bit depth and resolution that could be quite a lot of data.

Just sending triangle data could work, but that still leaves stuff like colouring/texturing the triangles for the Pokitto to handle.

i see thank you for this

so i think the best order of business is to get lua bytecode running and go from there

hardware wise i think connecting every posible pin to the poktto and esp32, spi,i2c,uart, maybe even all the gpio pins.
i think will stick with just the esp32 (no additional sd card) but having pas true pex for any additional hats?

1 Like

@Pharap oh also odd idea but what if i have a load function to enable the wifi and/or bluetooth functions from the c api i can add/open these functions to the function stack
(i hope that made sense)

It’ll probably use more memory than custom bytecode, but it’ll also probably be more flexible and easier to get up and running.

Lua is open source. Probably best to try with the standard C implementation before attempting to port eLua. The code can be grabbed from the download page or the lua github page.

Probably best to see what you need first.

Or maybe supply some kind of bypass mechanism so that e.g. SPI can have the slave select line work properly.
I think daisy-chaining is at least possible with SPI and I2C but there’s usually a small slow-down as a result.

Also I think before worrying about that you should decide which protocols are going to be the best for the job. Some protocols are faster than others and you might find that only some are fast enough to be useful.

I’m not completely sure what you mean.

I think you mean making the Pokitto code manually load the libraries on the ESP, in which case that won’t save flash space on the ESP, but it could possibly save RAM and it would probably keep the ‘separation of concerns’, so it would probably be a good idea.

1 Like

thank you for the help, i was already tinkering around with lua on pc in c
not sure if it can just run on esp32, not sure what the big projects are using (node-mcu)

im not entirly sure how to strip out just the bytecode interpreter

also my concern is more on ram then flash, i would like to avoid refrashing it to often

It should compile with just the standard C headers.


#include <assert.h>
#include <limits.h>
#include <stdlib.h>
#include <string.h>
#include <locale.h>
#include <math.h>
#include <stdio.h>
#include <errno.h>
#include <time.h>

The only one I can think might be an issue is stdio since a ‘headless’ board (i.e. no screen and no keyboard) usually has no concept of stdin or stdout.

I think the bit for compiling Lua to bytecode is just parser.h and parser.c, so getting rid of those would be the first thing to do.

After that you’d need to work through to replace the references to the parser with something that makes sense, and then get rid of the main function. There might be other bits that need doing as well.

Refreshing it?
Does it have some kind of read/write limit?

well from what i tried using arduino it took realy long to upload and had to check the flash several times , unlike the pokitto chip this thing can brick
the option for reflashing is there ofcource but its just something i wouldn’t want to deal with, im by no means an expert here

Doesn’t that mean the flash is the issue and not the RAM?

You said “my concern is more on ram then flash”, but that sounds like the opposite - flash/ROM is more of a concern than RAM.

Part of the reason it took so long is probably because there’s 448KB of ROM (i.e. flash). That’s actually quite a bit, and if you’re using UART it might be on a slow speed.

Assume it was running at 115200 baud (i.e. 115200 bits per second), that’s 14400 bytes per second and the flash is 458752 bytes so that’s 31.857 seconds to flash the whole thing. Obviously as you go down the scale (57600 baud, 38400 baud etc.) that starts to take a lot longer.

my concern is on optimizing for ram, i do not care about the flash in terms of user dousnt touch it
i know my wording is a bit complicated

and yes its very slow i dont recal the exact baud rate, but if the pokitto would have to do it and stream from flash it will take longer, also any interruption could brick it very badly

to go back to main subject im still looking for help on this project

I don’t get how it would brick it.
You’re only moving data into RAM, that doesn’t require ‘flashing’ and it shouldn’t brick your ESP unless you’re requesting too much or overwriting something you shouldn’t be somehow…

Are you using SPI or UART?

Right, I got a bit carried away with suggestions.

I can’t work on this full time because I’ve got other projects to finish first, so maybe I should just leave it at “if you need help/advice then PM me” or something?

if we went with reflashing the chip i meant using the pokitto as the programmer

i think the usb chip on the devboard is using uart

i looked around a bit more and there seems to be a fair amount of rtos “iot” type systems around for esp like zerynth vm wich is python, NodeMCU wich is lua, mongoose os wich i think is c++/js thing

Oh right.
Still not sure how that affects RAM though, as far as I’m aware RAM isn’t programmed when flashing the chip.

Flashing almost always uses UART.

NodeMCU looks closest to what you’ve suggested so far, though it might be a bit heavy considering it seems to need a file system in place.

Zerynth also looks good, especially the size aspect of it.
I particularly like that they have “embedded specific opcodes” and some of the changes they’ve made to the environment like removing compile and eval and not saving names.

I’m not too keen on Javascript so I’m slightly biased against Mongoose, but looking at mJS it seems they’ve removed a lot of the dynamic stuff and the JS code is only for development, it uses C++ for the actual production stuff.

Out of all of them I think Zerynth looks most appealing if it genuinely is as small as it claims to be, though I think NodeMCU’s examples look a lot easier than some of the Zerynth examples (though that could just be because I’m more used to Lua than Python).

My quick 2c on this. I would use the ESP as the main processor and Pokitto LPC11U68 as a GPU/APU. You can easily do all the rest through SPI.

1 Like