Saving user data on EEPROM & SD - development discussion

I agree, not just because you can avoid overwriting other people’s saves, but because memory cards can be quite large so you can have larger save states and you can then copy saves between Pokittos or onto a computer.

If EEPROM is going to be used for saving, I like this idea since it facilitates the copying/archiving/saving/restoring game saves that I mentioned earlier. It’s also optional.


Part of the problem was that most of the games had been developed before an EEPROM standard was discussed. The Pokitto has a bit of an advantage here because it’s still early days. To be honest I don’t know if anything was ever agreed upon, but it was hard to find something that suited everyone.

I think the Pokitto has an advantage here because of the SD slot.
EEPROM could be used for saving games, but I think the real power will come from being able to transfer those saves to the SD card. EEPROM has a fixed size and stays on the Pokitto, but SD cards can have very large capacities and files can be copied from SD cards to other devices, so it facilitates the ability to transfer saves and keep ‘archives’ of saves on a computer.

This is also a real plus for debugging games because players can send their saves (and maybe error logs) to the game developer.


I think maybe this ought to be more like highscore.token = Pokitto::SaveId("HISC"); so that any error comes from making an invalid SaveId rather than making the assignment operator do error checking.
Or if there’s a way to make the error come at compile time, that would be even better.

I like this idea especially.

In programmer parlance: serialisation.

A very good idea, but pointers could be a bit of an issue, there would have to be a way to make sure pointers are accounted for.
It’s a shame C++ doesn’t support reflection or that could have been handled with metadata/attributes.

3 Likes

I think abstracting it like that is a good idea, but we still need to decide on the best abstraction. We probably want to have a way to save a variety of data types without having to store metadata in EEPROM, to keep EEPROM usage as low as possible. Perhaps only a single tag for all the save data for the game, and the actual organization of the data is defined by the game itself.

I split this into its own topic, because I want to concentrate on this.

I propose that the number of datatypes is limited. Implementing a lot of different datatypes consumes program memory, because many different methods are/may be required.

So:

  • templated class
  • uint32_t and int32_t (naturally aligned to ARM Cortex memory space)
  • int16_t array (natural int datatype) of n*2 length
  • byte array of n*4 bytes length
  • string (is actually a n*4 byte array that is big enough to hold the string)
  • no pointers (?)
1 Like

I cut out the bits that aren’t related to SD/EEPROM to keep things focused.

1 Like

API looks good. It should cover the basic usage, and the byte array can be used to serialize a custom struct etc. As @Pharap mentioned there should be an error handling if the token is already in use or there is no space left. Should there be a delete function for a token too, which frees the memory?

1 Like

Come to think of it, I don’t know if the Pokitto uses exceptions or error codes for error handling.

Exceptions are more robust and flexible but can be harder for beginners to cope with and also increase code size/decrease performance.

Error codes are easier for beginners to understand but it can be a pain to constantly check error codes and can harm readability.

That’s a surprisingly tough question.

It depends how much ‘sandboxing’ there is going to be.

One solution is to hand the programmer an interface through which they can read and write their own save file but have no knowledge of anyone else’s save file. Kind of like handing someone a file writing object but not telling them about other files that are available.

e.g.

Pokitto::GameId gameId = Pokitto::GetId("My Game");
Pokitto::SaveWriter writer = Pokitto::CreateWriter(gameId);

writer.writeObject<LeaderBoard>(leaderboard);

writer.save();

And then the reverse:

Pokitto::GameId gameId = Pokitto::GetId("My Game");
Pokitto::SaveReader reader = Pokitto::CreateReader(gameId);

leaderboard = reader.readObject<LeaderBoard>();

Note that’s just a vague idea, the ‘game id’ concept needs refining, and maybe the notion of game slots would need refining too.
I like the idea of having a ‘GetId’ function so the Pokitto can check if the game exists or not and allocate some space if it doesn’t or something along those lines.
Preventing name clashes would still be an issue though.

1 Like

Is there an ETA on functions to save user data?
Only feature missing in Yatzy is the ability to save high scores.

1 Like

I doubt we’ll settle on something for a while yet.
Talks on this have taken a backseat to other things like the tutorials, getting Pokittos shipped and (I assume) getting the website for selling Pokittos up and running.

You can use the EEPROM part of the library for now though:

(Which I’ve only just become aware of and I must admit I’m impressed by the C++11-ness of it.)

I know that discussion on this has tapered off in this thread a bit, but there have been continued discussions about saving data in other topics and I wanted to propose a minor change for handling EEPROM.

As already discussed, if the loader can back up and restore EEPROM then saving data that way can be simple. I just want to also propose moving the data that the loader saves to the high addresses of the EEPROM instead of at the beginning. I would be happy to make these changes myself if there isn’t a problem with moving the saved loader data. Is the loader source available yet?

2 Likes

Its in PokittoEEPROM.h

I really think we need a saveblocks system, where a certain block of eeprom is marked unused / reserved and who owns it. Edit: each block indicates if there is another block in the chain or if it is the termination block of a chain. Very much like FAT file system

Was going to implement it but having problems finding time at the moment

PokittoEEPROM.h only provides the code for reading and writing EEPROM. I was asking about the loader, which I assume has the code for saving the current game’s name in EEPROM.

I agree that a blocks system could be a good abstraction layer, but for the lower-level api, I would prefer the settings saved by the loader to be at the end of EEPROM instead of at the start. Unless there is a reason that the data should be at the start, I personally feel that placing it at the end makes more sense, because it leaves the EEPROM as one contiguous region free for use, with only the high addresses reserved. It can be thought of as just a big save region that is free for the game to use without having to worry to start after the first N reserved bytes.

This is also why I feel that there is a disadvantage using blocks for saving. If a game needs to save a lot of data, it may need to use some or all of EEPROM (It could of course also save directly to the SD card). Also, if someone is playing a lot of games, they might run out of EEPROM space, and a game might either wipe out another game’s data or refuse to save. If the loader simply saves EEPROM contents to the SD card when switching games, then none of this is a problem; no games have to share EEPROM, and each game has more space to work with.

I propose a system of blocks with a passkey that is checked by the API.

API allocates a free block to the program, signs it with the passkey given by the author.

Other programs that ask for eeprom storage can not overwrite / reserve blocks to which they have no passkey

A “delete blocks” feature is added to the loader program.

Q1: how do you like the idea?
Q2: what would be a good passkey size & eeprom allocated block size?

Before you read the rest:
Should you or I move this to a different thread?

I wouldn’t want to hijack Armageddon’s thread.


Is your passkey idea like a unique identifier for each game, or an identifier given to each author, like REST APIs sometimes give?

I was thinking of a unique identifier for each game.
If you include the unique identifier in the checksum then it gives the checksum more robustness as well, since the rest of the data could look very similar for different games, but the game identifier (or ‘passkey’) should be unique for each game.

However there’s still several issues.

One issue is the block size.
Some games might need two blocks, and you might get cases where a game has to run into two blocks for the sake of just one byte.

Perhaps that could be mitigated by having several block sizes and encoding the size alongside the ‘passkey’ or something?

Obviously the block size should be a power of two.
Not just because powers of two are sacred and odd numbers are evil,
but also because it means we can make use of masking and bitwise operations.

Perhaps four block sizes, 128, 256, 512 and 1024?
Or maybe we need a 64 byte block or 32?
I’m not sure how much save space games use on average.

We could decree that anything that needs more than the maximum block size should use the SD card for the sake of other games.
(We have to consider that there might be games that need that much space for whatever reason.)

A ‘backup/restore blocks to/from SD’ would also be good, but it depends how much weight it adds to the loader.

Does the Pokitto have a limit on the number of times you can load a game?
I’m not entirely sure what the hardware is like, but I’ve become increasingly conscious of the upload limit on Arduino boards and I’m not sure if it’s the same for Pokitto.

Please do the honors.

1 Like

To bring a bit more context over:

If i look at the R4 cards that you can use with a Nintendo DS, they all use the sd to store a .SAV file where the savegames are stored.

Assuming nobody uses Pokitto without a sd card, this would be a really simple solution. Every game would have its own .sav file like : PokittoGame.sav on the sd

Example from a R4 card directory :slight_smile:

3 Likes

#Here is how EEPROM saving will work unless someone objects:

It’s time to stop going around in circles. I propose the following structure which I will make unless someone says it absolutely should not be done this way.

  1. Writing directly to EEPROM will be off limits unless you specifically unlock it (#define or some other mechanism). EEPROM write/read will check the lock and will return error is the lock is set

  2. Every programmer will specify a unique id code of 8 chars for their game. For example “ASTEROCK”

  3. There will be a base class called Pokitto::Cookie

  4. Cookie is an object that stores and reads data from EEPROM

  5. In your game/app you create a gamedata (or whatever name) class that inherits from Cookie

  6. Data, that you want to save permanently, for example Highscore, you will define inside your child class of Cookie

class gdata : public Cookie {
int highscore;
}

gdata gamedata;

7 . When you start your program, you register your cookie

gamedata.begin(“ASTEROCK”);

8 . The system checks if that key exists in the EEPROM key table, then calculates the space needed for your cookie from size of your cookie object. It then checks how many Crumbs are free in the EEPROM

9 . Each crumb is 32 bytes. There are 128 crumbs, but some are reserved for system settings and you can read them but not overwrite

10 . When your program key has been registered, and the system has found enough free crumbs to store your data, the system will register your key in the next free slot of the key table (48*8 bytes) and mark the crumbs reserved for your program in the crumb table (128 bytes)

11 . Important: key table and crumb table are also in the EEPROM, in an area that you can not access/overwrite

12 . you use the data objects inside the cookie as normal:

gamedata.highscore = player.currentscore; // It’s a new highscore!

13 . Finally you call the Cookie save method to write your data to EEPROM

gamedata.save; // save all data inside Cookie to EEPROM

14 . When you begin again the next time, the system now finds your key, the crumbs and loads the entire cookie as contiguous data back into your gamedata structure.

This OK with everyone so far? Saving EEPROM to SD is another thing and will be handled by the system as well but lets get this EEPROM saving working first.

4 Likes

I don’t entirely understand how the keys and the crumbs join up.
Having 48 keys in the key table but 128 crumbs in the crumb table seems a bit odd.
What are the values in the crumb table?
A key index?

In terms of implementation I think a template could possibly work better than inheritance.

class SaveData
{
public:
  int highScore;
};

Cookie<SaveData> cookie = Pokitto::acquireCookie<SaveData>("ASTEROCK");

I assume the writing is going to be something along the lines of:

template< typename Type >
class Cookie
{
private:
  std::uint32_t address;
  Type type;
public:
  void write(void)
  {
    const char * data = reinterpret_cast<const char *>(&type);
    write_eeprom(address, data, sizeof(Type));
  }
};

As for the other bits:
(For the sake of argument, ignore access modifiers).

class SaveKey
{
  constexpr const std::size_t bytesPerKey = 8;

  uint8_t values[bytesPerKey];
};

class SaveCrumb
{
  uint8_t value;

  uint8_t getValue(void) const
  {
    return (this->value & 0x7F);
  }
};

class SaveHeader
{
  constexpr const std::size_t keyCount = 48;
  constexpr const std::size_t crumbCount = 128;

  SaveKey keyTable[keyCount];
  SaveCrumb crumbTable[crumbCount];
};

The only outright complaint I have about that system is the name ‘crumb’.
All I can imagine from ‘crumb’ is a small piece of bread, which I don’t think makes a very good analogy.
Something like ‘block’ or ‘chunk’ would work better.
Otherwise it mostly sounds good (though I’m still not sure how the key-crumb relationship works.)


For what it’s worth, I was working on a similar idea earlier.
I’ll pitch it anyway in case there are aspects of both ideas that could be good:

  • EEPROM is divided into 32 byte ‘blocks’
  • An allocated block has a 16 byte heading (of which 9 bytes are used.*)
  • Each allocated block has a header consisting of:
    • A checksum field (4 byte/32 bit)
      • The checksum is calculated using all the header data, plus the entire contents of the allocated block(s)
    • A gameid field (4 byte/32 bit)
      • I’m not sure how this would be allocated, but it should be unique per game
    • A size field (1 byte/8 bit)
      • The value is the number of contiguous blocks allocated, e.g. a value of 8 is 8 blocks, which is 32 * 8 = 256 bytes

The blocks can be navigated by examining their size field and using that as the offset to the next block.
The first block (or two) is specifically for Pokitto system use.

Advantages:

  • Good use of the available space
  • Easy to understand
  • Built-in data integrity checking

Disadvantages

  • Allocating unique game ids would be hard
  • Traversing EEPROM might be slow
  • Can only traverse one way

* The remaining 7 are reserved for future modifications. That could be dropped to 12 bytes allocated, 9 used and 3 reserved if need be.

I didn’t get very far with making a code demo though:

#include <cstddef>
#include <cstdint>

namespace BlockSystem
{
	constexpr const uint32_t BytesPerBlock = 32;
	
	class SaveId;
	class BlockCount;
	class SaveBlockSettings;
	
	struct SaveId
	{
	private:
		std::uint32_t value;
		
	public:
		constexpr SaveId(uint32_t value)
			: value(value)
		{
		
		constexpr SaveId(char c0, char c1, char c2, char c3)
			: value(c0 << 24 | c1 << 16 | c2 << 8 | c3 << 0)
		{
		}
		
		SaveId(const char (&data)[4])
		{
			for(uint32_t i = 0; i < 4; ++i)
			{
				this->value <<= 8;
				this->value |= static_cast<uint8_t>(data[i]);
			}
		}
		
		operator uint32_t(void)
		{
			return this->value;
		}
	};

	struct BlockCount
	{
	private:
		std::uint8_t size = 0;
		
	public:
		constexpr BlockCount(void) = default;
		constexpr BlockCount(uint8_t size) : size(size) {}
		
		std::uint16_t getSizeInBytes(void) const
		{
			return static_cast<uint16_t>(size) * static_cast<uint16_t>(BytesPerBlock);
		}
		
		operator std::uint8_t(void) const
		{
			return this.size;
		}
	};

	struct SaveBlockSettings
	{
	public:
		std::uint32_t checksum;
		SaveId saveId;
		BlockCount size;
		
	private:
		constexpr const std::size_t targetSpace = 16;
		constexpr const std::size_t usedSpace = sizeof(checksum) + sizeof(saveId) + sizeof(size);
		constexpr const std::size_t reservedSpace = targetSpace - usedSpace;
		static_assert(targetSpace > usedSpace, "Error: SaveBlockHeader usedSpace exceeds targetSpace");
		
		// Pad it out to 16 bytes
		std::uint8_t reserved[reservedSpace];
	};
}
2 Likes

Ok, block it is then.

As for other comments:

  • mismatch between number of keys and blocks is because some games will use many blocks, some less

  • key and block are linked via the block table

  • my proposal is based on getting maximum storage out of the eeprom. If we use 16 byte header for 32 byte data, wouldn’t we be losing 1/3 memory for administration?

Still reading through your other ideas to understand it fully

2 Likes

As there are 48 keys i.e. save slots, that can run out at some point. Your propably do not have 48 active games but might have tested more that 48 games which write to EEPROM. So there should be a utility program (built in PokittoLib?) to remove unused keys.