User Tools

Site Tools


zxt:format

This is an old revision of the document!


ZXT Extended World Format

This is a draft! Things are still subject to change.

To discuss the draft, please log in and see the bottom of the documentation.

Introduction

Over the past thirty years, many ideas have been theorized and approaches tried with regards to extending and tweaking the behaviour and capabilities of the ZZT and Super ZZT game engines. With the release of the Reconstruction of ZZT greatly lowering the bar for creating such modified engine versions, the complexity and amount of forks, as well as interest in them, has begun to grow significantly. As such, in order to preserve to the extent possible the interoperability ZZT's ecosystem is known for, a standardized way to declare worlds which use or depend on extensions to the original ZZT and Super ZZT game engines is necessary.

A common format for declaring extensions will allow:

  • Creating forks and source ports of ZZT which support a more complete set of the diversified (non-ZZT 3.2-strict) game world catalog,
  • Mixing and matching extensions between forks at will,
  • Broader editor support for handling extended worlds.

The game world types that the ZXT format considers are as follows:

  • Creating completely custom experiences derived from ZZT or Super ZZT. Example: “The King in Yellow Borders”, WiL.
  • Creating games which can benefit from enhanced functionality, but are playable in unmodified ZZT or Super ZZT engines. Example: “Variety”, WiL.
  • Standardizing pre-fork games which modify ZZT or Super ZZT in an incompatible way. Example: “ZZT Enhancer”, Craig Boston; “Banana Quest”, WiL; 64K intros by bitbot.
  • Standardizing pre-fork games which modify ZZT or Super ZZT in partially or fully compatible ways. Example: “Angelis Finale: Episode 1”, Commodore (custom character set); “Daedalus' Obelisk”, Darren Hewer (optional ZZT modification to remove certain sounds and messages).

Definitions

These definitions may be used freely in extension standards without re-introduction.

  • The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this and following documents are to be interpreted as described in RFC 2119.
  • The following types, in data blob specifications, all little-endian (where applicable):
    • u8 - unsigned byte (8 bits),
    • s8 - signed byte,
    • u16 - unsigned word (2 bytes),
    • s16 - signed word,
    • u32 - unsigned dword (4 bytes),
    • s32 - signed dword,
    • u64 - unsigned qword (8 bytes),
    • s64 - signed qword,
    • bool - a special case of an u8, where zero means “false”, and non-zero means “true”,
    • pstring[X] - a Pascal-style string; a length u8 (N, of range between 0 and X) followed by X characters, of which the first N are considered valid.
    • pstring - a Pascal-style string; a length u8 (N, of range between 0 and 255) followed by N characters.
    • plstring - a Pascal-style long string; a length u16 (N, of range between 0 and 65535) followed by N characters.

Things marked “COMMENT:” below are “non-normative” and are not considered a part of the specification.

File Types

The extension data can be provided in one of two ways:

  • Providing a .ZAX file with the same filename as a world file (for instance, TOWN.ZZT and TOWN.ZAX) or board file, containing only the Extension Header.
  • Providing a .ZXT file, being formed as a concatenation of an Extension Header and a world file (cat TOWN.ZAX TOWN.ZZT > TOWN.ZXT is a valid operation) or board file.

This is done because neither solution is strictly superior:

  • .ZAX files can be used to extend games with additional functionality (keybind shortcuts, metadata, etc.) without the need to distribute a modified .ZZT file.
  • A .ZXT file cannot be understood by unmodified versions of ZZT or Super ZZT, which would require ZZT-compatible games making use of extensions to distribute two variants - a .ZZT and a .ZXT.
  • Distributing the world as a single .ZXT allows for greater user-friendliness than having to carry around two files, especially for non-ZZT-compatible worlds.

Notably, this does not cover save files. Saves are NOT guaranteed by the specification to be interchangeable between different implementations of the engine. (However, the authors expect a lot of implementations to simply follow ZZT's lead in terms of SAV files being valid ZXT files.)

Extension Header

The extension header's format is as follows:

Offset Type Name Description
0 u16 magic 0xF227 for ZZT-style worlds, 0xF527 for Super ZZT-style worlds, 0xB227 for ZZT-style boards, 0xB527 for Super ZZT-style boards.
2 u32 block_count The number of extension blocks which immediately follow.

After the header follow block_count extension blocks, in sequence:

Offset Type Name Description
0 u16 flags Extension flags; defined below.
2 u32 owner_id Extension owner ID
6 u16 selector_id Extension selector ID
8 u8 reserved_0 Reserved.
9 u16 field_length Field length. Can be between 0 and 65534; if set to 65535, an u32 containing the 32-bit field length follows.
11 u8[field_length] field_data Field data. Extension-defined.

An extension being “understood” is defined as one for whose ID pair the engine provides an implementation compliant with its standard.

The extension flags are as follows:

Bit Name Description Behaviour Layman's terms
0 parsing_must Required for parsing. If set, an implementation which does not understand this extension MUST NOT continue parsing of the extension block. Generally, this one will be set to 0.
1 reading_must Required for reading. If set, an implementation which does not understand this extension MUST NOT read the world file, but may continue parsing the extension block. If you're changing the world format in a breaking way, set this to 1.
2 writing_must Required for writing. If set, an implementation which does not understand this extension MUST NOT try to write the world file. If you're changing the world format in a partially-breaking way, set this to 1. (For instance, Variety redefines the flag section of a .ZZT file in a way which breaks reading only if at least two flags are set; however, a written world could have modified these flags, so we must prevent that. Another example would be re-using the padding fields.)
3 playing_should Recommended for playing. If set, an implementation which does not understand this extension SHOULD signal this to the end user if an attempt at playing the world is made. Non-strictly-breaking gameplay changes. For example, modified messages or sound effects. Sometimes, modified charsets or palettes.
4 playing_must Required for playing. If set, an implementation which does not understand this extension MUST signal this to the end user and prevent an attempt at playing the world. Breaking gameplay changes. New OOP commands if required, new element IDs, etc.
5 editing_should Recommended for editing. If set, an implementation which does not understand this extension SHOULD signal this to the end user if an attempt at editing the world is made. An engine does not have to make any effort to support such a world further. Non-format-breaking (format-breaking go into writing_must, reading_must or both) editor-side changes. Unknown element IDs count, for example.
6 preserve_should Recommended to preserve on resave. If set, an implementation SHOULD preserve the extension unchanged upon resave; if cleared, an implementation MUST NOT do so, and MUST discard the extension. Generally, you want this set to 1 - unless the field data relies on other board, world or extension data outside itself. Meant in particular for metadata.
7 vanilla_behavior Is this extension something ZZT 3.2 does anyway? If set, an implementation of ZZT 3.2 or Super ZZT 2.0 compliant with the considerations set out in the specification may be assumed to support the playing_must and playing_should conditions without further additional support. This is meant for extensions which impose additional restrictions upon a world not found in regular ZZT.
8 .. 15 reserved Reserved. If set, an implementation MUST NOT continue parsing of the extension block.

It is important to note that the flags can be distinct from the ID pair; for instance, the same ZZT-OOP extension can be defined as “recommended for playing” if optional for gameplay, but “mandatory for playing” if required for gameplay. However, an extension standard may require you to set or clear certain bits.

Interactions between multiple extensions

  • Extension Order: Extensions MAY depend on their order of definition in the file. Implementations MUST, as such, preserve the order of extension definition when writing a ZXT file.
  • Repeated Extensions: Extensions MAY allow being defined multiple times within a single list of extension blocks. However, if an extension does not specify behaviour in this event, the engine's reaction is undefined.

Any interactions not listed above are undefined behaviour and MUST NOT be relied upon. Here's an example scenario:

  • One extension is defined as “increasing the number of Keys from 7 to 16”,
  • Another extension is defined as “redefining the Keys to allow holding multiple at a time”,
  • It is not certain what should happen if both extensions are in place, unless additional detail is provided. One could interpret it as only allowing the seven vanilla ZZT Keys to be held multiple at a time, or as allowing all sixteen to do so.

Impact on world/board parsing

  • Header/Data Expansion: If an extension extends the board header, the world header, the tile data, the stat data, or any other existing in-file structure outside of the extension block, the order of parsing MUST be in ascending order of the 32-bit owner ID, then the 16-bit selector ID.
  • Incompatible Format Changes:
    • If any change which sets the reading_must flag to 1 is present, the implementation SHOULD set the world ID to 0xE227. Rare omissions are permitted, if the set of world format changes corresponds 1:1 to another non-standard world format ID (such as FreeZZT) - however, the list of extensions MUST reflect the same format changes. Extensions derive only from ZZT and Super ZZT's vanilla world formats.
    • If a ZXT or ZAX file is present, the implementation MUST ignore the world ID present in the file itself, and trust the ZXT/ZAX file instead.

Reserved extension owner IDs

  • The range 00000000 - 000000FF is reserved for standard extensions - expansions to this standard, as well as potential “obvious” changes everyone agrees upon.
  • The range FFFFF000 - FFFFFFFF is reserved for private extensions - usage when a standard for a given extension is not yet defined, or for internal/private experiments. Publicly released worlds SHOULD NOT use this range.

Considerations for .ZXT worlds and engines

Scope

  • Compliance in the context of ZXT support concerns only the game engine's operation. The user interface and other external factors not impacting gameplay or game presentation are not in scope.
  • An implementation MAY only be compatible with ZXT worlds containing certain extensions. Compatibility with un-extended ZZT 3.2 worlds is not a requirement.

Accuracy

  • An implementation consuming a .ZXT world, or a world with a .ZAX file, MUST NOT be expected to preserve ZZT's memory or stack layout exactly when running on a ZXT-compliant implementation.
    • Most out-of-bounds memory reads and writes, as such, are not guaranteed to work correctly.
    • The argTile2 trick pioneered in “Wake Up and Unlock The Door” counts as reliance on stack layout.
    • If you're relying on such behaviours, consider making an extension standard for it instead!
  • Certain behaviours have been grandfathered due to widespread usage pre-Reconstruction of ZZT. Extensions may change or disable their emulation. These are listed below:
    • Implementations MUST emulate the Black Key. The behaviour, including the gem value change and the message duration, MUST be preserved, although the specific text message emitted MAY be changed.
    • Implementations MUST support reading the state of stat -1. While the state of stat -1 in ZZT may change in rare circumstances, it is recommended to assume the following values:
Field Value
X 0
Y 1
Step X 256
Step Y 256
Cycle 256
P1 0
P2 1
P3 0
Follower 1
Leader 1
Under Element 1, Color 0x00
Data Pos 1
Data Len 1
  • A .ZXT world, or a world with a .ZAX file, MUST treat any feature which crashes regular ZZT (or causes double-frees, particularly double #BIND) as undefined behaviour, unless an extension states otherwise.
    • That is to say, ZXT-compliant implementations are free to fix them or not.

Unfinalized questions

These questions/concerns must be decided upon before the specification is finalized.

  1. Should extension definitions allow depending on their order of placement within the ZXT extension data list? This will complicate implementation, but allow additional flexibility. YES
  2. Should extension definitions allow for the same extension to be present multiple times in the same file? This will complicate implementation, but allow additional flexibility. YES
  3. Should there be any considerations on preserving extension information as part of save files? THE SPEC DOES NOT REGULATE SAVE FILES
    • Defining save files as implementation-specific is the easy way out and handy for some use-cases, but hinders save cross-compatibility and loading saves as game worlds in the editor.
    • Low-memory platforms may be hindered by requiring to load a lot of unnecessary extension data and keep it in memory. (For example, character sets.)
      • However, so long as the complete data is unnecessary to play the world, does it really hurt a save file?
      • Maybe we could allow a save file to be augmented with a .ZXT or .ZAX to restore blocks which are missing?
        • This would be best done with a “playing_mutates” bit - if it is 0, the extension could plausibly be restored from the original .ZXT file.
    • The compromise route, with no clear drawbacks, is to define save file rules as a SHOULD but not a MUST, allowing implementation-specific variants but encouraging a route for typical implementations.
  4. The use-case of ZAXes was defined as being strictly for worlds whose .ZZT file is compatible with ZZT 3.2. However, zzo38 pointed out that he'd prefer to use ZAXes even for worlds incompatible with ZZT 3.2. Should the ZXT format reserve magic world IDs for this scenario? PROPOSAL A
    • Proposal A:
      • Define a world ID (f.e. 0xE227) which means “this world file requires a ZXT header or ZAX file to be correctly parsed”,
      • Implementations must ignore the world ID if ZXT/ZAX information is present.
      • The drawback is that using a valid, but incorrect ZAX will break loading… but a valid, but incorrect ZAX is possible anyway (OOP changes, element changes, etc).
    • Proposal B:
      • Allow extensions to define acceptable world IDs.
      • Drawback: doesn't handle multiple world format changes
  5. How should extensions changing the world format (board header, world header, etc.) be handled?
    • Proposal A: In order of definition (see “allowing depending on order of placement”)
    • Proposal B: In ascending order of extension IDs
  6. Should there be more or fewer restrictions with regards to ZZT 3.2 behaviours allowed for a ZXT/ZAX-utilizing world? LOTS OF TWEAKS
    • The current ones are solely to not force implementations to emulate the entire ZZT 3.2 data segment; while Roton (commendably) tries to do it, it is a very large amount of effort that is also difficult to accomplish efficiently on some lower-end platforms.
  7. How should boards be prefixed? They can be of any size and don't have a magic number.
    • Proposal A: Boards are not defined by the standard.
      • Super ZZT and ZZT .BRD files use the same extension, but are not interchangeable. This wouldn't be new…
    • Proposal B: ZXT Boards start with two “0” bytes (a length of 0), immediately followed by the 0xB227 magic. A bit clunky, though…
  8. Which extensions should be stored as part of a .BRD file? How to handle conflicts on import/export? Do we need a new flag to check if an extension applies to a board's format?
  9. Should the specification make it clear that preference goes from last (most preferred) to first (least preferred) extension? (Think extensions which overwrite the same thing, but in increasingly more complex ways. For example, sound replacement could go from #PLAY → FM data → MIDI, or something.)

Implementations

Final specification

As the specification has not been finalized, no such implementations exist yet.

Draft specification

Comments

Discussion

Aaron Black, 2021/07/01 19:01, 2021/07/01 19:31

Comments about ZXT format; zxtsplit program (<1625029600.bystand@zzo38computer.org>; HTTP view).

Aaron Black, 2021/07/02 19:50
Should extension definitions allow depending on their order of placement within the ZXT extension data list?

I am not sure about this one (although the general consensus “yes” is probably best, since it is the general consensus, I think). Might it make sense to allow extensions to specify such things only for advisory rather than mandatory things?

Should extension definitions allow for the same extension to be present multiple times in the same file?

I believe there are good reasons to allow this. (Although, it depends on the specification of the specific extension; some might not define a meaning for such a case, while some will.)

Should there be any considerations on preserving extension information as part of save files?

There are two kind: portable save files and non-portable save files. Which an implementation uses is implementation-dependent. The specification is only relevant for portable save files. However, I think that saving a separate ZAX file is not so useful (and you can split them apart afterward if needed). My proposal is:

  • If the extension is enabled, then it MUST be retained, unless the specification of that extension specifies otherwise.
  • If the extension is not enabled, and the writing_must bit is set, then it MUST be discarded.
  • If the extension is not enabled, and the preserve_should bit is clear, then it MUST be discarded.
  • In any other case, the extension MAY be retained (unchanged) or discarded. It MAY allow the user to configure this.

I am thinking at the proposal mentioned in the document and the discord-of-zzt-bridge won't work. Here are my objections:

  • The playing_mutates bit doesn't seem helpful. If the extension is not implemented, then it can't mutate the extension anyways. If it is implemented, then it knows by the working of the specific extension if it can be mutated or not.
  • If extensions are reloaded from the original file when restoring a save game, then there might be the case that an extension is unexpectedly enabled or disabled, which might cause problems.

I also have an objection to my own proposal: Retaining extensions might also cause problems with being unexpectedly enabled or disabled. However, this might be fixed by adding a bit which requires an implementation supporting portable save files to alter the other flags (or discard the extension) when saving, as applicable. In this case, an implementation that does understand an extension sets the playing_must bit for that extension (and possibly others, if specified in the specification for that extension), and discards the extension if it is disabled. (Whether or not this is the case might depend on the individual game world, so that is why there would be a bit assigned for this purpose.)

Still, that doesn't solve the problem of low-memory situations, so that is another objection to this, too.

However, non-portable saves might just be simpler anyways, and not having portable saves.

zzo38 pointed out that he'd prefer to use ZAXes even for worlds incompatible with ZZT 3.2

Not quite. In such a case it is preferable to use a combined file, but for unforeseen reasons (possibly management of ZAX data), it might be necessary to split them, and possibly the files might get renamed without correct handling (this shouldn't happen, but just in case bad circumstances occur).

I agree with proposal A. The drawback mentioned is inevitable anyways, so you cannot really avoid that. Proposal B doesn't work for the reason described there (multiple world format changes), which is why I proposed the special world ID anyways.

How should extensions changing the world format (board header, world header, etc.) be handled?

My own proposal was to require the two orders to match for any extensions that affect world format (or that may otherwise conflict in other ways, if there are any “other ways”).

Aaron Black, 2021/07/03 05:37, 2021/07/03 18:39
Should there be more or fewer restrictions with regards to ZZT 3.2 behaviours allowed for a ZXT/ZAX-utilizing world?

There is the world and the engine. Most things not related to game state should be allowed to be freely differing in the implementation, although in some cases it should require such differences from standard ZZT to be documented. Some examples:

  • It may (but is not required to) have an option menu or configuration file to change key bindings or other input devices (such as a game pad).
  • An implementation SHOULD provide a way for the player to set/clear arbitrary flags by name by the use of some kind of debug/cheat prompt (it is OK if it is disabled by default, as long as the player can use some option to enable it; alternatively, it might be enabled by default when ZXT extensions are not present, or use some other criteria). If it doesn't work the same as standard ZZT, then this difference SHOULD be documented.
  • The restore game menu can use any sorting order and any user interface; it may display the timestamps of save games, screenshots, etc. How the player restores the game is also implementation-dependent, e.g. a quicksave/quickload like MegaZeux, pushing R only on the title screen to restore (like standard ZZT), pushing R on the title screen or during game play to restore (like standard Super ZZT), etc.
  • An implementation need not do keyboard handling in the same way as standard ZZT, although its effect on the game state must be effectively the same (even if the key mapping might be different). For example, it might not require a PLAYER or MONITOR to be present to save/restore/pause/quit the game, although pushing the arrows (or whatever other keys or other input devices are assigned to directional inputs) to move the player is required to work like in standard ZZT including in the presence of player clones (so attempting to move or shoot only has an effect if a PLAYER stat is present and it is its turn to execute).
  • The implementation MUST replace the stat 0 on the title screen with a MONITOR, although except for moving and shooting (and #IF ANY), the game engine need not otherwise care about the presence or absence of a MONITOR or PLAYER.
  • Other features separate from the game state such as message scrollback, tabbing to options, text search in popup texts, the speed of the animation for popup texts and for board transitions, sending the text to a printer or to a file, audio volume, etc, are allowed (but not mandatory).
  • The way that swap worlds are working might differ from standard ZZT, but this difference SHOULD be documented. (FreeZZT does not support swap worlds. See also the note below about “Wake Up and Unlock the Door”.)
  • The implementation of high scores might be different, or not implemented at all.

There is also the consideration of uninitialized memory reads, even if they are not out of bounds. This should also be considered as undefined behaviour by the specification, I think. An example is “Wake Up and Unlock the Door” by Dr.Dos. This stores a value in the argTile2 local variable of the OopExecute procedure, to be read later by a swap world. I am thinking that this variable is stored on the stack (as far as I can tell; I don't actually know how Turbo Pascal is compiling this as) and so will be uninitialized if the game depends on its value after the code that set it stops executing for the current frame. (Perhaps define an extension to specify if such behaviour is used.)

There is the possibility that I am wrong about some of the above; you can write your objections and/or agreements if you have any.

Adrian Siekierka, 2021/07/03 09:25

I agree with most of your points.

I do believe a lot of what you've suggested should be recommendations, however; that is, using the SHOULD and SHOULD NOT phrasing. There may be valid reasons to skip some of this functionality for very constrained ports, and ideally this would be documented. However, this spec should focus on the behaviour of the game engine and the guarantees of opting into the file format this causes; best standards and considerations for implementations might be best suited for another document.

The MONITOR replacement to me counts as emulating ZZT 3.2 behaviour, which is the baseline except for unstable/crash/data-segment-reliant behaviour already.

Limiting to the game engine is good; we already agreed saves are implementation-dependent. High scores, the user interface and other auxillary formats should be likewise.

Reliance on stack layout is unstable behaviour (I think there are situations in vanilla ZZT 3.2 during which OopExecute's stack can be overwritten - Scrolls?), and as such belong in the “undefined behaviour” pile like double free on #BIND.

Aaron Black, 2021/07/03 18:32

I changed the first MUST (about setting/clearing flags) to SHOULD now, since it does seem that it might be inappropriate for some implementations.

I agree that reliance on stack layout should be considered as undefined behaviour (unless, of course, an extension defines it).

Aaron Black, 2021/08/04 01:35, 2021/08/04 01:40

At the top, it mentions game world types that the ZXT format considers, but there are some others that it might also be applicable for, such as:

  • Adding metadata, such as catalog metadata
  • Games relying on unstable behaviour of the official ZZT executable
  • Games that are badly written and require more flags and/or stats than standard ZZT provides
  • Porting to more limited implementations, where the extension data can specify which things need to be implemented and how things can be omitted if not needed
You could leave a comment if you were logged in.
zxt/format.1625695988.txt.gz · Last modified: 2021/07/07 22:13 by asie