I doubt we’ll settle on something for a while yet.
Talks on this have taken a backseat to other things like the tutorials, getting Pokittos shipped and (I assume) getting the website for selling Pokittos up and running.
You can use the EEPROM part of the library for now though:
(Which I’ve only just become aware of and I must admit I’m impressed by the C++11-ness of it.)
I know that discussion on this has tapered off in this thread a bit, but there have been continued discussions about saving data in other topics and I wanted to propose a minor change for handling EEPROM.
As already discussed, if the loader can back up and restore EEPROM then saving data that way can be simple. I just want to also propose moving the data that the loader saves to the high addresses of the EEPROM instead of at the beginning. I would be happy to make these changes myself if there isn’t a problem with moving the saved loader data. Is the loader source available yet?
Its in PokittoEEPROM.h
I really think we need a saveblocks system, where a certain block of eeprom is marked unused / reserved and who owns it. Edit: each block indicates if there is another block in the chain or if it is the termination block of a chain. Very much like FAT file system
Was going to implement it but having problems finding time at the moment
PokittoEEPROM.h only provides the code for reading and writing EEPROM. I was asking about the loader, which I assume has the code for saving the current game’s name in EEPROM.
I agree that a blocks system could be a good abstraction layer, but for the lower-level api, I would prefer the settings saved by the loader to be at the end of EEPROM instead of at the start. Unless there is a reason that the data should be at the start, I personally feel that placing it at the end makes more sense, because it leaves the EEPROM as one contiguous region free for use, with only the high addresses reserved. It can be thought of as just a big save region that is free for the game to use without having to worry to start after the first N reserved bytes.
This is also why I feel that there is a disadvantage using blocks for saving. If a game needs to save a lot of data, it may need to use some or all of EEPROM (It could of course also save directly to the SD card). Also, if someone is playing a lot of games, they might run out of EEPROM space, and a game might either wipe out another game’s data or refuse to save. If the loader simply saves EEPROM contents to the SD card when switching games, then none of this is a problem; no games have to share EEPROM, and each game has more space to work with.
I propose a system of blocks with a passkey that is checked by the API.
API allocates a free block to the program, signs it with the passkey given by the author.
Other programs that ask for eeprom storage can not overwrite / reserve blocks to which they have no passkey
A “delete blocks” feature is added to the loader program.
Q1: how do you like the idea?
Q2: what would be a good passkey size & eeprom allocated block size?
Before you read the rest:
Should you or I move this to a different thread?
I wouldn’t want to hijack Armageddon’s thread.
Is your passkey idea like a unique identifier for each game, or an identifier given to each author, like REST APIs sometimes give?
I was thinking of a unique identifier for each game.
If you include the unique identifier in the checksum then it gives the checksum more robustness as well, since the rest of the data could look very similar for different games, but the game identifier (or ‘passkey’) should be unique for each game.
However there’s still several issues.
One issue is the block size.
Some games might need two blocks, and you might get cases where a game has to run into two blocks for the sake of just one byte.
Perhaps that could be mitigated by having several block sizes and encoding the size alongside the ‘passkey’ or something?
Obviously the block size should be a power of two.
Not just because powers of two are sacred and odd numbers are evil,
but also because it means we can make use of masking and bitwise operations.
Perhaps four block sizes, 128, 256, 512 and 1024?
Or maybe we need a 64 byte block or 32?
I’m not sure how much save space games use on average.
We could decree that anything that needs more than the maximum block size should use the SD card for the sake of other games.
(We have to consider that there might be games that need that much space for whatever reason.)
A ‘backup/restore blocks to/from SD’ would also be good, but it depends how much weight it adds to the loader.
Does the Pokitto have a limit on the number of times you can load a game?
I’m not entirely sure what the hardware is like, but I’ve become increasingly conscious of the upload limit on Arduino boards and I’m not sure if it’s the same for Pokitto.
Please do the honors.
To bring a bit more context over:
If i look at the R4 cards that you can use with a Nintendo DS, they all use the sd to store a .SAV file where the savegames are stored.
Assuming nobody uses Pokitto without a sd card, this would be a really simple solution. Every game would have its own .sav file like : PokittoGame.sav on the sd
Example from a R4 card directory 
#Here is how EEPROM saving will work unless someone objects:
It’s time to stop going around in circles. I propose the following structure which I will make unless someone says it absolutely should not be done this way.
Writing directly to EEPROM will be off limits unless you specifically unlock it (#define or some other mechanism). EEPROM write/read will check the lock and will return error is the lock is set
Every programmer will specify a unique id code of 8 chars for their game. For example “ASTEROCK”
There will be a base class called Pokitto::Cookie
Cookie is an object that stores and reads data from EEPROM
In your game/app you create a gamedata (or whatever name) class that inherits from Cookie
Data, that you want to save permanently, for example Highscore, you will define inside your child class of Cookie
class gdata : public Cookie {
int highscore;
}gdata gamedata;
7 . When you start your program, you register your cookie
gamedata.begin(“ASTEROCK”);
8 . The system checks if that key exists in the EEPROM key table, then calculates the space needed for your cookie from size of your cookie object. It then checks how many Crumbs are free in the EEPROM
9 . Each crumb is 32 bytes. There are 128 crumbs, but some are reserved for system settings and you can read them but not overwrite
10 . When your program key has been registered, and the system has found enough free crumbs to store your data, the system will register your key in the next free slot of the key table (48*8 bytes) and mark the crumbs reserved for your program in the crumb table (128 bytes)
11 . Important: key table and crumb table are also in the EEPROM, in an area that you can not access/overwrite
12 . you use the data objects inside the cookie as normal:
gamedata.highscore = player.currentscore; // It’s a new highscore!
13 . Finally you call the Cookie save method to write your data to EEPROM
gamedata.save; // save all data inside Cookie to EEPROM
14 . When you begin again the next time, the system now finds your key, the crumbs and loads the entire cookie as contiguous data back into your gamedata structure.
…
This OK with everyone so far? Saving EEPROM to SD is another thing and will be handled by the system as well but lets get this EEPROM saving working first.
I don’t entirely understand how the keys and the crumbs join up.
Having 48 keys in the key table but 128 crumbs in the crumb table seems a bit odd.
What are the values in the crumb table?
A key index?
In terms of implementation I think a template could possibly work better than inheritance.
class SaveData
{
public:
int highScore;
};
Cookie<SaveData> cookie = Pokitto::acquireCookie<SaveData>("ASTEROCK");
I assume the writing is going to be something along the lines of:
template< typename Type >
class Cookie
{
private:
std::uint32_t address;
Type type;
public:
void write(void)
{
const char * data = reinterpret_cast<const char *>(&type);
write_eeprom(address, data, sizeof(Type));
}
};
As for the other bits:
(For the sake of argument, ignore access modifiers).
class SaveKey
{
constexpr const std::size_t bytesPerKey = 8;
uint8_t values[bytesPerKey];
};
class SaveCrumb
{
uint8_t value;
uint8_t getValue(void) const
{
return (this->value & 0x7F);
}
};
class SaveHeader
{
constexpr const std::size_t keyCount = 48;
constexpr const std::size_t crumbCount = 128;
SaveKey keyTable[keyCount];
SaveCrumb crumbTable[crumbCount];
};
The only outright complaint I have about that system is the name ‘crumb’.
All I can imagine from ‘crumb’ is a small piece of bread, which I don’t think makes a very good analogy.
Something like ‘block’ or ‘chunk’ would work better.
Otherwise it mostly sounds good (though I’m still not sure how the key-crumb relationship works.)
For what it’s worth, I was working on a similar idea earlier.
I’ll pitch it anyway in case there are aspects of both ideas that could be good:
checksum field (4 byte/32 bit)
gameid field (4 byte/32 bit)
size field (1 byte/8 bit)
32 * 8 = 256 bytesThe blocks can be navigated by examining their size field and using that as the offset to the next block.
The first block (or two) is specifically for Pokitto system use.
Advantages:
Disadvantages
* The remaining 7 are reserved for future modifications. That could be dropped to 12 bytes allocated, 9 used and 3 reserved if need be.
I didn’t get very far with making a code demo though:
#include <cstddef>
#include <cstdint>
namespace BlockSystem
{
constexpr const uint32_t BytesPerBlock = 32;
class SaveId;
class BlockCount;
class SaveBlockSettings;
struct SaveId
{
private:
std::uint32_t value;
public:
constexpr SaveId(uint32_t value)
: value(value)
{
constexpr SaveId(char c0, char c1, char c2, char c3)
: value(c0 << 24 | c1 << 16 | c2 << 8 | c3 << 0)
{
}
SaveId(const char (&data)[4])
{
for(uint32_t i = 0; i < 4; ++i)
{
this->value <<= 8;
this->value |= static_cast<uint8_t>(data[i]);
}
}
operator uint32_t(void)
{
return this->value;
}
};
struct BlockCount
{
private:
std::uint8_t size = 0;
public:
constexpr BlockCount(void) = default;
constexpr BlockCount(uint8_t size) : size(size) {}
std::uint16_t getSizeInBytes(void) const
{
return static_cast<uint16_t>(size) * static_cast<uint16_t>(BytesPerBlock);
}
operator std::uint8_t(void) const
{
return this.size;
}
};
struct SaveBlockSettings
{
public:
std::uint32_t checksum;
SaveId saveId;
BlockCount size;
private:
constexpr const std::size_t targetSpace = 16;
constexpr const std::size_t usedSpace = sizeof(checksum) + sizeof(saveId) + sizeof(size);
constexpr const std::size_t reservedSpace = targetSpace - usedSpace;
static_assert(targetSpace > usedSpace, "Error: SaveBlockHeader usedSpace exceeds targetSpace");
// Pad it out to 16 bytes
std::uint8_t reserved[reservedSpace];
};
}
Ok, block it is then.
As for other comments:
mismatch between number of keys and blocks is because some games will use many blocks, some less
key and block are linked via the block table
my proposal is based on getting maximum storage out of the eeprom. If we use 16 byte header for 32 byte data, wouldn’t we be losing 1/3 memory for administration?
Still reading through your other ideas to understand it fully
As there are 48 keys i.e. save slots, that can run out at some point. Your propably do not have 48 active games but might have tested more that 48 games which write to EEPROM. So there should be a utility program (built in PokittoLib?) to remove unused keys.
A suggestion for the Cookie Manager program:
With such a manager, it’s almost a file system with tiny path haha
Linked how though?
Do the 7 bits of a crumb/block table entry indicate which key they correspond to?
E.g. if crumbTable[4] == 5 does that correspond to the crumb being owned by the key at keyTable[5]?
In your system, are blocks allocated contiguously or can a game have its blocks spread out?
Not quite.
In my system all the allocated blocks are contiguous and a game can claim more than 32 bytes.
So if a game claims 32 bytes (1 block), the header size means they only get 16 usable, but if they claim 64 bytes (2 blocks), they get 48 usable. If a game claimed 512 bytes (16 blocks), there would be the constant 16 bytes for the header and the remaining 496 bytes would be usable etc
Maybe this will make it clearer:
checksum (4 bytes)
id (4 bytes)
size (1 byte)
reserved (9 bytes)
data (((size * 32) - 16) bytes)
So to find a block by traversing the allocated blocks:
EeepromPointer findBlock(SaveId saveId)
{
EepromPointer pointer = pointerToFirstByte;
while(isValid(pointer))
{
SaveBlockSettings settings;
EEPROM.read(pointer, settings);
if(settings.saveId == saveId)
return pointer;
pointer += settings.size.getSizeInBytes();
}
return invalidEepromPointer;
}
Most likely an id of 0 would be used to indicate an unused block.
Hopefully we will have a system for backing up entries to SD card so you can treat the EEPROM as your active data and the SD card as your archived data.
Or manually to organise before loading a new game.
Essentially that kind of is what we’re doing.
These ideas are similar to how file systems usually work.
@jonne’s proposal is similar to the FAT family of file systems because it uses a table to track the block entries. Mine is closer to an algorithm that’s sometimes used to implement malloc (it’s more of a bare bones linked list with offsets instead of pointers).
After looking at FAT, one other idea I had was to have a table that stores the game id, an index and a size, with index and size being multiples of the block size.
struct TableEntry
{
std::uint8_t gameId[6];
std::uint8_t index;
std::uint8_t size;
};
Then it could fit 64 entries in 512 bytes (or 42 for the extra 2 bytes in the name).
This would probably be a much easier system to perform reconfiguring operations on.
I still don’t understand why people would rather use EEPROM than a file on SD…
Because EEPROM is cool.
To enable to save the state if Pokitto do not have an SD card at all.
FYI EEPROM savegame blocks system is almost working already
Going to finish it up tonight