Interpreted language (esp32 specific)

im working on a a couple of projects (rpg, esp32) and looking at the python thing
and im wondering if it might be better/easier to write a custom interpreted language

main ideas are:

  • compact, as light as it can go
  • dynamic loading and unloading of chunks
  • extendible with custom function calls

this that im not sure about but may be necessary:

  • stack based
  • limited types (8bit int/char, 16bit floats, 256 sized arrays)

im not an expert on this, im badicly thinking of interpreted bytecode thats general purpose
im not sure how many instructions i would need for this but basic aritmetic loading chunks of sd card and saving to sd card seem essential

1 Like

Have you tried getting Lua going?

It woulsn’t support dynamic loading without modification, but it’s extensible and has a very simple bytecode and language.

Have a read of “A No-Frills Introduction to Lua 5.1 VM Instructions”:

ANoFrillsIntroToLua51VMInstructions.pdf (218.7 KB)

Even if it doesn’t sell you on Lua, it’ll give you an insight into how an existing VM works in terms of bytecode.


Have you found a library for that?
The typical floating point format is very confusing.

That’s potentially very wasteful.
You’d be better off with length-prefixed/length-tracked arrays instead.

Stack based is easier for code generation but can get tricky if there’s a lot of variables to manipulate.

Perhaps a hybrid - a stack with a limited number of local variables?


The big question is: static typing or dynamic typing?

i wish i could use lua, but elua has no suport for m0+ chips due to memory concerns and in personal experiments compiled lua is rather large wich bites any use out of it

lua and python probebly aswel are going to have a hard time on mode15 games where the buffer is holding a massive chunk of memory

have not looked into floats yet, and im not super sure how needed they are, if i could trow them out and just run on 8 bit integers it optimises allot

im sorry i did not mean a chuck of 256 bytes but a max size of 256 (due to running everything in 8bit intigers) else there not adressable but your right a length prefix would be prefreble

im not sure but im thinking about a stack and then an vector of arrays for the arrays wich there you could store all your locals

i know that sounds very how you do it and perhaps terrible to use but im realy trying to be compact in memory
it could really use a small higher level language ontop to be more user friendly

im not sure

im thinking of it having 2 programs or “chunks” a main or manager program
and secondary that gets loaded/unloaded

the main program calls the secondary, and the secondary calls main
alowing to swap both programs out if necessary
i think the stack and array memory should be unafected between program swapping so may need manual cleanup in the programs

What size does it tend to compile to?

My version compiled for x64 Windows comes to just 158KB for the .dll.
The interpreter that links to that is just 12KB.
The compiler is 114KB, but I think that’s doing a lot of extra stuff.

With some effort it could probably be made smaller.
Especially by using some C++ techniques to replace some of the heavier C bits.

Either that or if eLua is usually small, that could be adapted.
It’s usually easier to adapt something that works than to start from scratch.

Frankly so would most bytecode-based/interpreted languages.
There will be things that you can’t do with a bytecode-based language that you could with machine code.
Comparing the two won’t get you very far.

Or you could use fixed points.

It would probably be better to have a max size of 255 (otherwise the size has to be stored in two bytes rather than one).

Otherwise that should be fine.

Arrays would need some form of dynamic allocation if they’re going to be variably sized.
In which case you may as well just treat them as a size and a pointer.

I’m not entirely sure whether a std::vector would be the right approach, that would need some testing to see what works best.

Dynamic is less flexible, but can often consume more RAM.
Static is often better optimised for space and size, but makes the compiler harder to make.

What would be the purpose of each program?


Some other suggestions:

Pawn

Pawn is a simple, typeless, 32-bit extension language with a C-like syntax.
The Pawn compiler outputs P-code (or bytecode) that subsequently runs on an
abstract machine. Execution speed, stability, simplicity and a small footprint
were essential design criterions for both the language and the abstract
machine.

Squirrel

Squirrel is a high level imperative, object-oriented programming language, designed to be a light-weight scripting language that fits in the size, memory bandwidth, and real-time requirements of applications like video games.

its just a tought on how to manige a chunk based program the idea is one is the maniger and loader thats going to be longer in memory and the secondary are more like functions that get dealocated
it alows a cleanup phase or setup phase. this might be a bad aproch

i had a idea of larger programs taking to long an having it preload the second function in chunks wile still running the game and logic with the first program. but i think im making it complicated

any experiance with pawn or Squirrel i have seen them around but unsure how they work or how they implement

i should probebly mention at the moment for testing purposes i have tinybasic running on the esp32
one idea is to strip that down into single character commands (something similar i think sinclair did for there basic on the zxspectrum)

I’m still not entirely sure what the idea is supposed to be.

Either way, an easier approach would be to just have a concept of a bytecode function (a function implemented entirely in bytecode),
then track how often each function is used and when you need to load a new one, unload the one that has been used the least.

It’s not easy, but it should be effective.

Programs, especially games, tend to spend a large amount of time in their main loops and end up with a lot of functions that aren’t used often.

Nope, I’ve only heard of them, never used them.
I think Pawn is smaller though.

well the options i see on lower end micro’s (arduino) seems to be forth and tinybasic

i was intending no sort of functions but maybe theres something there
we can break down the file on the sd card into functions and load/unload the manually
having automatic unload seems like a bad idea since it will result in random slowdowns where having manual control can be acounted for "loading screens or a transition between scene like in an rpg going from overworld to a battle)

Unless you’re planning on going the same route as Espruino (ie, no bytecode at all, parse the Javascript on the microcontroller) then you need to break this into two separate problems: A VM that runs on the microcontroller, and the compiler (that runs on your PC?) which generates the bytecode for the VM.

I’m not sure what the advantage of doing this is, compared to simply writing the code in C++ or using a transpiler (a compiler that takes, say, Basic and translates that to C++).

If you are going to use a bytecode, you can use some techniques common to emulators, so you can look at some of those to see how they work.
For efficiency, I’d suggest having about 4 512-byte blocks of code that you can read from the SD at once. Don’t worry about function boundaries, they can be arbitrarily large and have both hot and cold parts inside. When you need to execute a block that hasn’t been loaded, you discard the one that hasn’t been used recently and read a new one over it. The tendency is for “hot” code to stay in RAM while things that are not important will get discarded.

Don’t assume 8-bit data types are better, ARM processors really like 32 bits.

so we are just talking about the pokitto side and its “compiled” the top level language is currently not my concern maybe we can compile down from an existing language but im not sure

if only we could execute c/c++ compiled code on ram …

i mean the idea is that the program (or better a sub program) runs in ram instead of flash like specific logic for special events or one time things in longer games, like rpg’s

might be true that arm prefers 32 but we are starved on the ram side

If we’re starved on RAM, how would executing code on RAM help?
Is the reason you want an interpreter just to get around the RAM limitation by doing memory swapping in the VM?

Forth’s quite good.
It’s kind of hard to program in if you aren’t used to reverse polish notation, but it’s reasonably decent.

(I wrote a pseudo-forth interpreter once.)

You will want functions.
Functions result in less memory use overall.

But how do you decide when to unload them?

At least with the ‘automatic’ approach, the unloading is based on a metric - the frequency of use.
That’s (sort of) how JIT compilers work, resulting in slow execution the first time around but speed up with later execution.

  • Not having to change the contents of flash.
  • Possibly doing some scripting on the Pokitto itself
  • Fun
  • Learning

32 bit values would decrease flash usage, but 8 bit values are better for decreasing RAM usage.
RAM is probably the more important thing here.

@FManga
the reason for interpreter is mostly for the flash actualy, rpg project is rapidly running out of memory and have to ofload logic for specific combat events, quests, and “cutscenes”
on the other hand the esp32 project we cant reflash the esp32 (easly) so sending bytecode up to be executed seems like the best option, the idea here is to make it sort of general purpose and reusable for any other project that might need this

1 Like

To be honest I think the RPG and ESP projects would be better off using different languages.

ESP would be better off using something generic (like pawn, lua etc) so it can be used as a coprocessor and can be more flexible.

The RPG would be better off using something very specific to the game engine. If you have opcodes that equate to game commands like “give player money” and “open text box” and “close text box”, you’ll end up with smaller code (look into Pokemon scripting opcodes and Mother 3 opcodes.)

i guess your right, though nether is really solved.
on the esp side its been a bit of a struggle getting programs on it, so far only arduino version seem to be stable and easy to use, but i cant run or modify the lua (i think its an rtos) esp32 project
like i said only got tinybasic working on it, wich i geuss is somthing

on the rpg side, i dont realy know what to do with it apart from a giant statemachine and maybe a small text parser.
wich kinda sucks since i would probebly have to do this all over again for a metroidvania or card game

another probably really bad idea is streaming bytecode from the sd card
but thats something im already doing with the mapdata in rpg project and well, it eventualy softlocks the game

Better than nothing.
If that can register custom functions then that might be worth a go.

No need to parse text.
Just store it directly as bytecode.

Use either a switch statement or a std::map to handle the different opcodes. (Whichever uses less memory - though std::map is more flexible.)

You could keep all the opcodes relating to text and simple calculations, but you’d probably have to alter stuff like “give money” and “give monster”.

You could try to make it more generic if you want, but that will come at a cost, probably code size.
One option would be to reserve a batch of opcodes for game-specific use, make the bytecode engine handle all the builtin codes itself and then for the game-specific ones, have the games register a callback function with the engine so it can pass those specific opcodes to the callback for the game to handle.

Also think carefully about your design.
If you’re having a text engine, you want a class that acts as a generic textbox interface but can either have its implementation swapped out or can use different graphics settings so it can draw itself differently depending on the game.

If you mean not doing any buffering then yes.
Your best option is to have map files and have all the scripts needed for the map loaded along with the map if possible.

Either that or load the entire script before running it.
If the scripts are small then it shouldn’t be an issue.

1 Like

I was refering to a scenario where bytecode is on the SD card, without a compiler in flash. You wouldn’t be able to do scripting on the Pokitto. You’d have to do compiling on a PC, then transfer to the Pokitto. The workflow becomes quite similar to compiling native code.