User Tools

Site Tools


zxt:format

ZXT Extension Format Specification (Draft)

This page is for draft work! For the latest official version of the ZXT specification, please check this page.

Version ?.?.?

By Adrian “asie” Siekierka

Special thanks: endgame, GreaseMonkey, Lancer-X, Noser, The Mysterious KM, WiL, zzo38

The specification follows a MAJOR.MINOR.PATCH numbering scheme, where:

  • Changes to the MAJOR number are expected to introduce breaking changes, resulting in a change of the extension header magic number;
  • Changes to the MINOR number are expected to introduce changes which don't break backwards compatibility, such as giving meaning to reserved values or defining undefined behaviour;
  • Changes to the PATCH number are expected to introduce changes which don't impact the meaning of the specification, such as fixing typos or clarifying used language.

For example, a ZXT 1.1.x world which doesn't utilize functionality specific to ZXT 1.1.x would be expected to work as-is in a ZXT 1.0.x engine implementation; there is no such requirement placed on a ZXT 2.0.x world.

Introduction

Extending and tweaking the functionality of the ZZT game engines 1) has always been an undercurrent in their world development community. While many games relying on edited executables or TSRs 2) have been released in the past, they ran into key problems, keeping their count small and the overall idea unpopular:

  • The lack of source code greatly increased the difficulty of performing non-trivial modifications.
  • The requirement for special batch scripts or modified executables, typically made specifically for a given game, created additional hassle for the end user when trying to play such a game.
  • No consistent means of signaling required extensions was devised. For most worlds, this made them indistinguishable from ZZT 3.2-compatible ones for the end user or reimplementations, hurting compatibility.

However, the release of the Reconstruction of ZZT in 2020 greatly lowered the bar for creating such modified engine versions, solving the first problem. With it, interest in creating game-specific forks and providing enhanced functionality has reappeared. Nonetheless, the remaining issues persisted and, as such, a standardized way to declare engine extensions utilized by a given world was deemed necessary.

The following use cases were considered:

  • Creating forks and source ports of ZZT which can support any set of extensions at will,
  • Mixing and matching extensions from various creators within worlds,
  • Usage of extensions within typical ZZT game operations (playing, saving, editing).

The following categories of worlds were considered:

  • Completely custom experiences derived from ZZT game engines. Example: “The King in Yellow Borders”, WiL.
  • Creating games which can benefit from enhanced functionality, but are playable in unmodified ZZT game engines. Example: “Variety”, WiL.
  • Standardizing pre-fork games which modify ZZT or Super ZZT in an incompatible way. Example: “ZZT Enhancer”, Craig Boston; “Banana Quest”, WiL; 64K intros by bitbot.
  • Standardizing pre-fork games which modify ZZT or Super ZZT in ways which are technically compatible. Example: “Angelis Finale: Episode 1”, Commodore (custom character set); “Daedalus' Obelisk”, Darren Hewer (optional ZZT modification to remove certain sounds and messages).

Coverage

The specification covers:

  • The format of extension header data and resulting world files,
  • Rules for processing extension data and handling identified edge cases,
  • Expectations placed upon game engines implementing the specification.

Notably, the specification currently does NOT cover:

  • The format of save (.SAV) files. We have decided not to mandate this at this time.
  • The format of board (.BRD) files. This is due to the niche nature of the subject, and is likely to be expanded upon in a future version of the specification.

Definitions

These definitions may be used freely in extension standards without re-introduction.

  • The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this and following documents are to be interpreted as described in RFC 2119.
  • The following types, in data blob specifications, all little-endian (where applicable):
    • u8 - unsigned byte (8 bits),
    • s8 - signed byte,
    • u16 - unsigned word (2 bytes),
    • s16 - signed word,
    • u32 - unsigned dword (4 bytes),
    • s32 - signed dword,
    • u64 - unsigned qword (8 bytes),
    • s64 - signed qword,
    • bool - a special case of an u8, where zero means “false”, and non-zero means “true”,
    • pstring[X] - a Pascal-style string; a length u8 (N, of range between 0 and X) followed by X characters, of which the first N are considered valid.
    • pstring - a Pascal-style string; a length u8 (N, of range between 0 and 255) followed by N characters.
    • plstring - a Pascal-style long string; a length u16 (N, of range between 0 and 65535) followed by N characters.

Where the specification mentions the terms ZZT game engine or .ZZT, one may substitute Super ZZT game engine and .SZT respectively.

File Format

Attachment

The extension data can be provided in one of two ways:

  • Providing a .ZAX file with the same filename as a world file (for instance, TOWN.ZZT and TOWN.ZAX) or board file, containing only the Extension Header.
  • Providing a .ZXT file, being formed as a concatenation of an Extension Header and a world file (cat TOWN.ZAX TOWN.ZZT > TOWN.ZXT is a valid operation) or board file.

The two approaches are made with specific intents in mind, but may be used in any way the creator sees fit:

  • The intent of .ZAX files is to extend games with additional functionality (keybind shortcuts, metadata, etc.), while preserving compatibility with the ZZT game engine. (They can also serve as patches on top of an otherwise unmodified .ZZT file.)
  • The intent of .ZXT files is to contain a modified ZZT game within one world file. This is especially useful for games which cannot be supported by unmodified versions of the ZZT game engine.

If a .ZXT file is being loaded, a .ZAX file should be ignored - .ZAX files apply for .ZZT files, not .ZXT files.

The behaviour of multiple concatenated extension headers contained within one .ZXT or .ZAX file is undefined; distributed .ZXT or .ZAX files MUST NOT rely on it.

Extension Header

The extension header's format is as follows:

Offset Type Name Description
0 u16 magic 0xF227 for ZZT-style worlds, 0xF527 for Super ZZT-style worlds, 0xB227 for ZZT-style boards, 0xB527 for Super ZZT-style boards.
2 u32 block_count The number of extension blocks which immediately follow.

After the header follow block_count extension blocks, in sequence:

Offset Type Name Description
0 u16 flags Extension flags; defined below.
2 u32 owner_id Extension owner ID.
6 u16 selector_id Extension selector ID.
8 u8 reserved_0 Reserved; must be set to 0.
9 u16 field_length Field length. Values between 0 and 65534 refer to the length in bytes; if set to 65535, an u32 containing the 32-bit field length in bytes follows.
11 u8[field_length] field_data Field data. Extension-defined.

An extension being “understood” is defined as one for whose ID pair the engine provides an implementation compliant with its standard.

The extension flags are as follows:

Bit Name Description Behaviour Layman's terms
0 parsing_must Required for parsing. If set, an implementation which does not understand this extension MUST NOT continue parsing of the extension block. Generally, this one will be set to 0.
1 reading_must Required for reading. If set, an implementation which does not understand this extension MUST NOT read the world file, but may continue parsing the extension block. If you're changing the world format in a breaking way, set this to 1.
2 writing_must Required for writing. If set, an implementation which does not understand this extension MUST NOT try to write the world file. If you're changing the world format in a partially-breaking way, set this to 1. (For instance, Variety redefines the flag section of a .ZZT file in a way which breaks reading only if at least two flags are set; however, a written world could have modified these flags, so we must prevent that. Another example would be re-using the padding fields.)
3 playing_should Recommended for playing. If set, an implementation which does not understand this extension SHOULD signal this to the end user if an attempt at playing the world is made. Non-strictly-breaking gameplay changes. For example, modified messages or sound effects. Sometimes, modified charsets or palettes.
4 playing_must Required for playing. If set, an implementation which does not understand this extension MUST signal this to the end user and prevent an attempt at playing the world. Breaking gameplay changes. New OOP commands if required, new element IDs, etc.
5 editing_should Recommended for editing. If set, an implementation which does not understand this extension SHOULD signal this to the end user if an attempt at editing the world is made. An engine does not have to make any effort to support such a world further. Non-format-breaking (format-breaking go into writing_must, reading_must or both) editor-side changes. Unknown element IDs count, for example.
6 preserve_should Recommended to preserve on resave. If set, an implementation SHOULD preserve the extension unchanged upon resave; if cleared, an implementation MUST NOT do so, and MUST discard the extension. Generally, you want this set to 1 - unless the field data relies on other board, world or extension data outside itself. Meant in particular for metadata.
7 vanilla_behavior Is this extension something ZZT does anyway? If set, an unmodified implementation of ZZT 3.2 or Super ZZT 2.0 may be assumed to support the playing_must and playing_should conditions without further additional support. This is meant for extensions which either (a) impose additional restrictions on the engine implementation not present in regular ZZT, or (b) depend on ZZT 3.2/Super ZZT 2.0 implementation details which are not required for an otherwise ZXT-compliant implementation. The intent is for external tooling to be able to declare a world ZZT 3.2-compatible.
8 .. 15 reserved Reserved. If set, an implementation MUST NOT continue parsing of the extension block.

It is important to note that the flags can be distinct from the ID pair; for instance, the same ZZT-OOP extension can be defined as “recommended for playing” if optional for gameplay, but “mandatory for playing” if required for gameplay. However, an extension standard may require you to set or clear certain bits for compliance.

Extension IDs

Extension IDs are allocated on a per-owner ID basis. Within one owner ID, selector IDs SHOULD be used in a “one per extension” manner.

Owner ID Ranges

  • The range 00000000 - 000000FF is reserved for standard extensions - types of changes everyone agrees upon, as well as potential expansions to this standard.
  • The range 00000100 - FFFFFEFF is available for public extensions - you are free to claim them via signaling intent on the ZXT Owner IDs page.
  • The range FFFFFF00 - FFFFFFFF is reserved for private use extensions - prototypes and/or experiments within internal/development versions of worlds and engine implementations. Publicly released worlds and engine implementations SHOULD NOT use this range.

Interactions

Cross-Extension Interactions

  • Extension Order: Extensions MAY depend on their order of definition in the file. Implementations MUST, as such, preserve the order of extension definition when writing a ZXT file.
  • Repeated Extensions: Extensions MAY allow being defined multiple times within a single list of extension blocks. Implementations MUST apply the extensions in order; that is to say, a later extension MAY override the settings of an earlier extension.

Any interactions not explicitly defined and not part of any enabled extension's specification are undefined behaviour and MUST NOT be relied upon. For example:

  • One extension is defined as “increasing the number of Keys from 7 to 16”,
  • Another extension is defined as “redefining the Keys to allow holding multiple at a time”,
  • It is not certain what should happen if both extensions are in place, unless additional detail is provided. One could interpret it as only allowing the seven vanilla ZZT Keys to be held multiple at a time, or as allowing all sixteen to do so.

World Format

  • Header/Data Expansion: If an extension extends the board header, the world header, the tile data, the stat data, or any other existing in-file structure outside of the extension block, the order of parsing MUST be in ascending order of the 32-bit owner ID, then the 16-bit selector ID.
  • Incompatible Format Changes:
    • If any change which sets the reading_must flag to 1 is present, the implementation SHOULD set the world ID to 0xE227. Rare omissions are permitted, if the set of world format changes corresponds 1:1 to another non-standard world format ID (such as FreeZZT) - however, the list of extensions MUST reflect the same format changes. Extensions derive only from ZZT and Super ZZT's vanilla world formats.
    • If a ZXT or ZAX file is present, the implementation MUST ignore the world ID present in the file itself, and trust the ZXT/ZAX file instead.

Engine Extension Scope

  • ZXT standard compliance concerns only the game engine's operation. The user interface and other external factors not impacting gameplay or game presentation are not in scope.
  • An implementation MAY choose to be compatible only with ZXT worlds containing certain extensions. Compatibility with un-extended ZZT worlds for a ZXT implementation is OPTIONAL.

Engine Accuracy

Engines are expected to be accurate to the ZZT game engine's intended behaviour. However, preserving ZZT's memory and stack layouts - and the respective bugs - are OPTIONAL, with the following exceptions:

  • Implementations MUST emulate the Black Key. The behaviour, including the gem value change and the message duration, MUST be preserved, although the specific text message emitted MAY be changed.
  • Implementations MUST support reading the state of stat -1. While the state of stat -1 in ZZT may change in rare circumstances, it is recommended to assume the following values:
Field Value
X 0
Y 1
Step X 256
Step Y 256
Cycle 256
P1 0
P2 1
P3 0
Follower 1
Leader 1
Under Element 1, Color 0x00
Data Pos 1
Data Len 1

Note that any feature which crashes regular ZZT (or causes double-frees, particularly double #BIND) remains undefined behaviour. ZXT-compliant implementations may choose to fix them.

Of course, any of the assumptions in this section can be overridden by an extension.

Best Practices

This is not a formal part of the specification - it's more of an “advice” section.

World Developers

  • Distributing the world as a single .ZXT allows for greater user-friendliness than having to carry around two files, especially for non-ZZT-compatible worlds.
  • Once you opt into the .ZXT format, ZZT tricks relying on memory/stack layout are not guaranteed to work. If you're relying on such behaviours, consider making an extension standard for them instead! This means in particular, but not limited to:
    • Most out-of-bounds memory reads and writes,
    • The argTile2 trick pioneered in “Wake Up and Unlock The Door“.

Extension Specification Authors

  • As soon as your extension is used by publicly-released worlds or implemented in publicly-released implementations, the expectation is for it not to change in a breaking manner! While undefined or explicitly reserved behaviour may be clarified further, breaking changes should be introduced via new extensions. There are two ways to avoid that issue during development:
    • Use an ID inside the area of private use extensions while the extension's shape is not yet fully formed,
    • Mark your public specification as “Draft” while details are not fully decided upon.
  • If your extension is intended to work only with ZZT or Super ZZT, but not the other, it's a good idea to mention that explicitly in the specification.

Engine Developers

  • Not every ZXT-compliant engine implementation must provide every extension! Don't fall neck-deep into code bloat, especially on your first try. Focus on the ones you care about, or which games you enjoy utilize.
  • Remember that your save files need to be aware of which extensions ought to be enabled in order to correctly restore a gameplay session. An easy way for this is to do what ZZT itself does, and share the code for creating world files and save files.

Side Notes

This is not a formal part of the specification - rather, it provides insight into some contested decisions.

  • I am aware that the “owner/selector ID” system is not ideal. However, other alternatives considered had flaws deemed more important:
    • A linear ID allocation system would most likely cause conflicts;
    • A string-based ID system would increase implementation difficult, particularly on older compilers;
    • The owner/selector divide was introduced after the decision to faciliate 48-bit IDs on older compilers, particularly Turbo Pascal. While 32-bit IDs were considered, they were rejected as potentially being too constraining.
  • Save files are not mandated by the specification for two key reasons:
    • Some extensions may require data specific to save files, and implementations may wish to have the freedom to decide how to store them in this case;
    • Some implementations may store ZZT data in a custom format internally, and removing this requirement can make their code simpler.
  • Multiple concatenated extension headers may be used as a feature in the future - for example, for faciliating multi-patch application for ZZT worlds.

Future Suggestions

ZXT 2.0

ZXT 1.1

  • Unified way of handling external file dependencies (for example, large music files).
    • Approach A: Add a bit which signifies that the data of an extension block is actually the filename of the file containing the real data.
    • Approach B: Add a single extension with a list of external filenames used by the world.
    • Approach C: Add a set of extensions to allocate “asset IDs” or “asset names”, which may point to in-file data or out-of-file assets.
    • Approach D: Do nothing.
    • Only approach A would need to be discussed as part of a spec update.
  • Add a bit which signifies “if this extension is not supported, the immediately following extension may be used as its substitute.”
    • Rationale: For example, multiple charset sizes (8×14, 6×8, 4×6, etc.) for different platforms.

Comments

1)
ZZT and Super ZZT
2)
Terminate-Stay Resident programs

Discussion

Aaron Black, 2021/07/01 21:01, 2021/07/01 21:31

Comments about ZXT format; zxtsplit program (<1625029600.bystand@zzo38computer.org>; HTTP view).

Aaron Black, 2021/07/02 21:50
Should extension definitions allow depending on their order of placement within the ZXT extension data list?

I am not sure about this one (although the general consensus “yes” is probably best, since it is the general consensus, I think). Might it make sense to allow extensions to specify such things only for advisory rather than mandatory things?

Should extension definitions allow for the same extension to be present multiple times in the same file?

I believe there are good reasons to allow this. (Although, it depends on the specification of the specific extension; some might not define a meaning for such a case, while some will.)

Should there be any considerations on preserving extension information as part of save files?

There are two kind: portable save files and non-portable save files. Which an implementation uses is implementation-dependent. The specification is only relevant for portable save files. However, I think that saving a separate ZAX file is not so useful (and you can split them apart afterward if needed). My proposal is:

  • If the extension is enabled, then it MUST be retained, unless the specification of that extension specifies otherwise.
  • If the extension is not enabled, and the writing_must bit is set, then it MUST be discarded.
  • If the extension is not enabled, and the preserve_should bit is clear, then it MUST be discarded.
  • In any other case, the extension MAY be retained (unchanged) or discarded. It MAY allow the user to configure this.

I am thinking at the proposal mentioned in the document and the discord-of-zzt-bridge won't work. Here are my objections:

  • The playing_mutates bit doesn't seem helpful. If the extension is not implemented, then it can't mutate the extension anyways. If it is implemented, then it knows by the working of the specific extension if it can be mutated or not.
  • If extensions are reloaded from the original file when restoring a save game, then there might be the case that an extension is unexpectedly enabled or disabled, which might cause problems.

I also have an objection to my own proposal: Retaining extensions might also cause problems with being unexpectedly enabled or disabled. However, this might be fixed by adding a bit which requires an implementation supporting portable save files to alter the other flags (or discard the extension) when saving, as applicable. In this case, an implementation that does understand an extension sets the playing_must bit for that extension (and possibly others, if specified in the specification for that extension), and discards the extension if it is disabled. (Whether or not this is the case might depend on the individual game world, so that is why there would be a bit assigned for this purpose.)

Still, that doesn't solve the problem of low-memory situations, so that is another objection to this, too.

However, non-portable saves might just be simpler anyways, and not having portable saves.

zzo38 pointed out that he'd prefer to use ZAXes even for worlds incompatible with ZZT 3.2

Not quite. In such a case it is preferable to use a combined file, but for unforeseen reasons (possibly management of ZAX data), it might be necessary to split them, and possibly the files might get renamed without correct handling (this shouldn't happen, but just in case bad circumstances occur).

I agree with proposal A. The drawback mentioned is inevitable anyways, so you cannot really avoid that. Proposal B doesn't work for the reason described there (multiple world format changes), which is why I proposed the special world ID anyways.

How should extensions changing the world format (board header, world header, etc.) be handled?

My own proposal was to require the two orders to match for any extensions that affect world format (or that may otherwise conflict in other ways, if there are any “other ways”).

Aaron Black, 2021/07/03 07:37, 2021/07/03 20:39
Should there be more or fewer restrictions with regards to ZZT 3.2 behaviours allowed for a ZXT/ZAX-utilizing world?

There is the world and the engine. Most things not related to game state should be allowed to be freely differing in the implementation, although in some cases it should require such differences from standard ZZT to be documented. Some examples:

  • It may (but is not required to) have an option menu or configuration file to change key bindings or other input devices (such as a game pad).
  • An implementation SHOULD provide a way for the player to set/clear arbitrary flags by name by the use of some kind of debug/cheat prompt (it is OK if it is disabled by default, as long as the player can use some option to enable it; alternatively, it might be enabled by default when ZXT extensions are not present, or use some other criteria). If it doesn't work the same as standard ZZT, then this difference SHOULD be documented.
  • The restore game menu can use any sorting order and any user interface; it may display the timestamps of save games, screenshots, etc. How the player restores the game is also implementation-dependent, e.g. a quicksave/quickload like MegaZeux, pushing R only on the title screen to restore (like standard ZZT), pushing R on the title screen or during game play to restore (like standard Super ZZT), etc.
  • An implementation need not do keyboard handling in the same way as standard ZZT, although its effect on the game state must be effectively the same (even if the key mapping might be different). For example, it might not require a PLAYER or MONITOR to be present to save/restore/pause/quit the game, although pushing the arrows (or whatever other keys or other input devices are assigned to directional inputs) to move the player is required to work like in standard ZZT including in the presence of player clones (so attempting to move or shoot only has an effect if a PLAYER stat is present and it is its turn to execute).
  • The implementation MUST replace the stat 0 on the title screen with a MONITOR, although except for moving and shooting (and #IF ANY), the game engine need not otherwise care about the presence or absence of a MONITOR or PLAYER.
  • Other features separate from the game state such as message scrollback, tabbing to options, text search in popup texts, the speed of the animation for popup texts and for board transitions, sending the text to a printer or to a file, audio volume, etc, are allowed (but not mandatory).
  • The way that swap worlds are working might differ from standard ZZT, but this difference SHOULD be documented. (FreeZZT does not support swap worlds. See also the note below about “Wake Up and Unlock the Door”.)
  • The implementation of high scores might be different, or not implemented at all.

There is also the consideration of uninitialized memory reads, even if they are not out of bounds. This should also be considered as undefined behaviour by the specification, I think. An example is “Wake Up and Unlock the Door” by Dr.Dos. This stores a value in the argTile2 local variable of the OopExecute procedure, to be read later by a swap world. I am thinking that this variable is stored on the stack (as far as I can tell; I don't actually know how Turbo Pascal is compiling this as) and so will be uninitialized if the game depends on its value after the code that set it stops executing for the current frame. (Perhaps define an extension to specify if such behaviour is used.)

There is the possibility that I am wrong about some of the above; you can write your objections and/or agreements if you have any.

Adrian Siekierka, 2021/07/03 11:25

I agree with most of your points.

I do believe a lot of what you've suggested should be recommendations, however; that is, using the SHOULD and SHOULD NOT phrasing. There may be valid reasons to skip some of this functionality for very constrained ports, and ideally this would be documented. However, this spec should focus on the behaviour of the game engine and the guarantees of opting into the file format this causes; best standards and considerations for implementations might be best suited for another document.

The MONITOR replacement to me counts as emulating ZZT 3.2 behaviour, which is the baseline except for unstable/crash/data-segment-reliant behaviour already.

Limiting to the game engine is good; we already agreed saves are implementation-dependent. High scores, the user interface and other auxillary formats should be likewise.

Reliance on stack layout is unstable behaviour (I think there are situations in vanilla ZZT 3.2 during which OopExecute's stack can be overwritten - Scrolls?), and as such belong in the “undefined behaviour” pile like double free on #BIND.

Aaron Black, 2021/07/03 20:32

I changed the first MUST (about setting/clearing flags) to SHOULD now, since it does seem that it might be inappropriate for some implementations.

I agree that reliance on stack layout should be considered as undefined behaviour (unless, of course, an extension defines it).

Aaron Black, 2021/08/04 03:35, 2021/08/04 03:40

At the top, it mentions game world types that the ZXT format considers, but there are some others that it might also be applicable for, such as:

  • Adding metadata, such as catalog metadata
  • Games relying on unstable behaviour of the official ZZT executable
  • Games that are badly written and require more flags and/or stats than standard ZZT provides
  • Porting to more limited implementations, where the extension data can specify which things need to be implemented and how things can be omitted if not needed
You could leave a comment if you were logged in.
zxt/format.txt · Last modified: 2021/08/10 15:00 by asie