As could be seen in the previous thread, there are many different and useful things that the Loader could do: Flash python scripts, flash GB ROMs directly, show thumbnails, long file names, author names, licences, etc.
The only problem is space. Whatever space the loader takes, it takes away from games.
So a custom loader should…
take up less space
do more stuff
be fully compatible with the current bins and loader
be easy to install and revert to the vanilla loader
be quick, simple, fancy, look good, bake a cake…
This still just a bunch of concepts I’m experimenting with, but here’s what I’ve been thinking about.
TL;DR:
When activated, the loader will execute “boot.pex” from the SD card.
PEX being a “Pokitto EXecutable”. That’s about it, really.
Now for a slightly more detailed overview…
Pokitto Executable/DLL File Format - (.POK? .PEX? .PDL?)
The file format is composed of a series of sections. In general, they are all optional, can be in any order, and must start with a 4-byte type id and a 4-byte length.
Section Types:
0 - Padding. Used for aligning the following section to a 512-byte boundary.
1 - Index. Acts as a directory for quickly finding other sections without having to scan the entire file. Should be the first section, ideally.
2 - Code Offset Address. Defines what address the following Code section should be copied to.
0x10008000 - Code. Only section that has no length after the type id, therefore must be last.
3 - Offset & Code. Contains offset address and code. Can appear multiple times.
4 - Library Entry Point. Libraries have no NVIC, the Entry Point should return a struct with function pointers.
5 - Code CRC. Used to detect if the game is already in Flash.
6 - HAT compatibility [hat name][flag]. Where flag indicates: 0-incompatible, 1-compatible/optional, 2-required (won’t work without the hat)
7 - long name (text)
8 - author (text)
9 - description (text)
10- screenshot image, 110x88, 2bpp
11- title logo image, 100x20, 2bpp
12- icon image, 24x24, 2bpp
The Loader
In order to free up flash, it is desirable that the loader be as small as possible. I’d like to reduce it to the bare minimum: system initialization, FS API, PEX parser API. Much like the ROM APIs built-in to the processor, the FS and PEX APIs will be made available to games.
This means we won’t need to have two PFFS implementations (the loader’s and the game’s) in flash all the time. Games in the PEX format can safely assume the FS API will be available, since they can’t be loaded using the Flash button.
After basic initialization, the loader is capable of only one thing: loading boot.pex. If you rename your game to that it will get copied to flash as soon as you “Press C for Loader”, but that’s not very useful. Unlike BINs, PEX files can specify exactly where code should be loaded. Not just flash, code can be loaded to RAM (almost instantly). If you enter the loader then reset the Pokitto, the last game will still be there. This also allows for code to be loaded in run-time like a DLL (.PDL?).
Boot.pex will use the FS API present in flash to scan (a directory in) the SD card to find plugins in the form of PDLs. These would do the actual loading. It would then be possible to open various different file types: bmp, txt, bin, hex, py, gb, movies…
In the case of GB files, for example, the parser would first copy the emulator (if it isn’t already present) and then copy the GB ROM into it. No online converter needed. I imagine something similar would work with Python. A BMP/Movie loader wouldn’t actually flash anything, it would just show the BMP.
Some details intentionally vague since it’s too early to know for sure. Others, because I’m still trying to figure them out, like how to draw the UI in a plugin or the bit about baking cake.
cool name but wouldent that be confusing with the pex extension header (io pins)?
anyway the thing you want do seems like a single file to parse and load
i know pffs dousnt suport it but wouldent a “app container” (a .pok folder with the bin, images, save files, and your discriptor file) be easier?
that discriptor could just have a flag wich utility it wants to use if its not a binary
program
it be cleaner if also had an utility folder
this loader you sugest dous mean you have to have an sd card to boot the device
Possibly… Nothing is set in stone yet, it’s just a “working name” until somebody thinks of something better.
Heh, sounds like something Apple would do.
It’s certainly possible to use a folder to store everything. It’s just a different kind of container. I feel that using a single file would be cleaner, easier to distribute and less error-prone (it’s harder for the end user to mess up a single file than a directory).
So, for GB ROMs, each one would need to be in an individual folder with its own descriptor, each pointing to the emulator in the utility folder. Wouldn’t it be easier for the end user if the loader could be extended to support custom files?
You’d need an sd card to run the loader, which only makes sense with a card anyway. You’d be able to boot the game normally without a card.
EDIT: Another thing I want to do with this idea is to move all the UI code out of the loader to free up space and to allow different UIs (skins).
yea i mean i made pokitto.app for the mac so dont sue me XD it works and is a clean look for the user, also potential to contain the file io a little since developer owns that folder and can create whatever “save.txt” file they want witout acidently overiding someone else
the thing then is the loader should know all the file stypes and that may grow in time, just having the name of the binary that needs to be preloaded would make more sence in my opinion, you might have multiple visual novel engines and the game code might both be a txt file or something, this could
i defenetly agree with that, i would even like to see the pokitto loge to be just loaded from sd card over always bieng on flash, though i think jonne wont like that much
thing that comes to mind for me would be a Pokitto Interpreted Language thats super lightweight to parse.
i have been talking about this before a couple times and had been experimenting with this ( i was calling it .PIL) but i never gotten anything great working, though @Pharap always said stuff about bytecode interpreters
it could all be streamed in with pffs and with maybe only one draw comand (dirrectdraw) could make the file navigator and more
That’s something I’m trying to avoid. I want to put loaders for all these extra file types on the SD card as separate plugins. The result could be something like this:
Each icon in the horizontal line would be a loader plugin. You can have an arbitrary amount of plugins. Each expands into a list of files that it can open. You can have more than one loader capable of reading a certain file type.
This could be done by an interpreter, as you suggested, but it would take up far more space in flash than simply loading native code.
do you know of a way to execute from ram?
my concern is if ech time you start the device your overiding the flash multiple times (overiding bootloader, overiding with file system, overiding with emulator or game)
I think having a different extension (.pex is fine by me or .pop or .pog - short for pokitto program or pokitto game, or whatever) is the right way to go
As soon as I saw the new topic I was kind of expecting to see a working bin here … but I know we have all been spoilt by your skillz so far
I am not sure if I understand your main idea fully, but this how I think it works:
Boot.pex shows the list of files. It can show icon, screenshot, etc. too if the file is *.pex, right(?). For other files, e.g. *.py, *.bmp, it just shows the default icon, if anything.
When you select a list item that is a pex file, Boot.pex flashes it and makes a reset(?). After that the pex file can be run. Normal Pokitto binary files can be converted to pex files in PC, and put to SD card.
When you select a list item that is e.g. bmp file, Boot.pex searches for the PDL that can handle the file, flashes it and the bmp file to rom(no reset needed?), and gives control to the pld program. The pld then shows the bmp image
Does this sound about correct?
MicroPython can either load a py or mpy(bytecode) file to ram, or use frozen rom file (bytecode). The frozen rom file can be executed in-place, no need to load it to ram.
In theory, just copy code to RAM and call it. I’m still experimenting with this, the tricky part is making useful code that fits strict size limits and doesn’t clash with other libraries (bss sections can’t overlap). Right now I’m making a screen mode library that fits in SRAM1 (2kb).
The flash will be written to less than it is now. The only time it writes to flash is once you pick a game. If you select a GB ROM, for example, it will check if the emulator is already in flash before copying it. In that case it would only copy the ROM itself. Once that is done, it will reset and it won’t write to flash again.
Hehe, I think I’ll go with .pop.
It sure is weird making a topic with a bunch of unproven concepts and no bin to show. I just wanted to make sure this is something the community would like before investing too much effort into it.
Kinda.
boot.pex will search for the pdls it can use and it lets the user select one. So the first thing you see will be a menu like this: [Python] [Gameboy] [Games] [Text] [Images]. Each being a separate pdl.
Once you select a loader, boot.pex will load that PDL into RAM and search for files that it can read.
For each file, boot.pex will request an icon/name/screenshot from the selected loader. If there is none, the loader can return a default.
Once you select a file, boot.pex will request the loader PDL to load it.
– In the case of a BMP, I was thinking of not writing anything to flash here. The Image.pdl would simply display the image instead.
– when you select a PEX file, game.pdl flashes it and resets. There is no special treatment for pex files in boot.pex, they’re loaded by a pdl just like the other files. Yes, pex files are made in a PC and copied to the SD. A regular bin is a valid pex, though the inverse might not be true.
– when you select an MPY, both it and micropython are copied to flash, then it resets.
can you clearafy what you mean by that, in theory you dont hardly need any ram for screen mode if you draw directly to screen from sd card (direct 4color bitmap for fonts and sprites) ok maybe the a small amount for the pffs buffering stuff
(or just go with 565 sprites (16bit per pixel) would be slower but more colorfull since it could show all preview images in there games pallets)
maybe unrealated but on the interpreted side could you do anything practical with something like a brainfuck thing? that would be very small on the flash?
#include <stdlib.h>
char m[9999], *n[99], *r = m, *p = m + 5000, **s = n, d, c;
main()
{
for (read(0, r, 4000); c = *r; r++)
c - ']' || (d > 1 ||
(r = *p ? *s : (--s, r)), !d || d--), c - '[' || d++ ||
(*++s = r), d || (*p += c == '+', *p -= c == '-', p += c == '>',
p -= c == '<', c - '.' || write(2, p, 1), c - ',' || read(2, p, 1));
}
Like I said, I’m experimenting and nothing is set in stone yet. I haven’t decided on which screen mode we’ll actually use. We can use direct mode, but that limits what we can do with animations. I’ll let someone better with art propose a UI mockup before deciding anything.
I'm actually testing a bytecode interpreter, but it's not meant to be turing-complete.
That bytecode does all the system and LCD setup, leaving things in a ready-to-go state for the rest of the app. Still not sure if it’s a good idea, I haven’t compared it to the current C++ implementation to see if it actually saves space. All I know is: it works. The final bin does everything in system_11U6x.cpp, initializes SPI, initializes the LCD, then writes to the screen in mode 1 (code not included above) while taking up about 1.5kb in total.
Absolutely. I cannot figure out any downsides compared to a current implementation, but just huge advantages
iirc it is not currently supported, but I think it is not hard to make it to support. The whole python program is just a huge string. However, in that case MP needs to compile the script to the bytecode and save it to ram (including bitmaps etc.), so we are much limited by the free ram size (MP interpreter needs quite much stack too).
I think it depends whether you want plugins to be able to do arbitrary execution or not.
If you wanted to limit plugins to a subset of commands and speed wasn’t a concern then bytecode might be a viable approach.
I think .plp - Pokitto Loader Plugin.
The interpreter would be tiny but the programs would be unrealisitcally huge.
If an esoteric language was going to be used, something stack-based like FALSE would be a better choice.
That’s 1KB for the original x86 compiler, which also does zero error handling, so you can’t go by that.
The size of the code ported to ARM would be different, especially if we added a safety net to prevent it doing dangerous things.
A bytecode variation of FALSE would be pretty easy though, I write them for fun when I get bored :P
Technically FALSE is already bytecode-based,
and the instructions it uses are conviniently also printable characters.
I actually already have a basic stack-based bytecode language implemented because I was going to use it as part of a Pokitto project I didn’t get round to doing because I never got round to ordering that 3.3v-5v bridge.
It would need some modifying to be applicable to this loader though.
My intent in the design is, in order to minimize flash usage, whenever possible, everything on flash should serve more than one purpose:
The FS API will be used by the loader and games that support it.
The initialization routine, which is in a sort of bytecode, could also be used by the loader and games. The interpreter for that is just 32 thumb instructions.
The PEX API will be used by the loader to load plugins and to load the actual games. It can also be used by games that want dynamic code loading. If no games want that, at least the parser can be trivially small.