Arcane knowledge
These are some absurd quirks and facts we’ve learned about in the process of making GDPatch. While most of these aren’t really useful to know, they’re quite amusing to keep around for storytelling!
GDScript’s parser is wacky
Section titled “GDScript’s parser is wacky”We observed several bugs in GDScript’s tokenizer and parser when porting it into Rust for GDPatch.
Some Godot versions write garbage into the header
Section titled “Some Godot versions write garbage into the header”When introducing bytecode script support into Godot 4.3, the header array size was improperly set, so 4 bytes of the bytecode header were filled by uninitialized memory. This surfaced as GDScript binary files being non-deterministic and was fixed in Godot 4.5.
Vector<uint8_t> contents;contents.resize(20); // <-- this is 20 bytes...encode_uint32(identifier_map.size(), &contents.write[0]);encode_uint32(constant_map.size(), &contents.write[4]);encode_uint32(token_lines.size(), &contents.write[8]);// and there's nothing written to offset 12 here!encode_uint32(token_counter, &contents.write[16]);Godot reads garbage if a script starts with a tab
Section titled “Godot reads garbage if a script starts with a tab”If a file begins with a tab, the file pointer will underflow, causing the tokenizer to read before the start of the buffer. This causes the tokenizer to read garbage data and spew several errors.
Godot freaks out if you escape a carriage return
Section titled “Godot freaks out if you escape a carriage return”When escaping a carriage return (\r) in a string without a line feed (\n) following it, Godot reads null characters and spews several errors. In this example, there is a carriage return between the \ and . characters.
GDScript identifiers are XORed
Section titled “GDScript identifiers are XORed”In the binary GDScript format, all identifiers are inexplicably XORed with the key 0xB6. This is presumably a (very weak) attempt at obfuscation. (source)
// ...for (int i = 0; i < len; i++) { uint8_t tmp[4]; encode_uint32(s[i], tmp);
for (int b = 0; b < 4; b++) { contents.write[buf_pos + b] = tmp[b] ^ 0xb6; }
buf_pos += 4;}// ...cargo test will load GDPatch
Section titled “cargo test will load GDPatch”Because GDPatch is implemented as a dynamic library that’s loaded into the game process, when cargo test goes to build and run the crate, it ends up triggering the loader entrypoint. This means GDPatch appears in the build logs, which is quite amusing to stumble upon for the first time.
I have discovered the hard way that if you write a dynamic library to be injected into another process with LD_PRELOAD that it will also be loaded when using `cargo test`
GDScript has no official grammar
Section titled “GDScript has no official grammar”GDScript’s parser is implemented in a multi-thousand-line C++ file, and there are no official specifications or grammars for the language. The Godot docs used to contain a EBNF grammar, but it is “descriptive only” and the engine does not use it. (GDScript is also not easily described in an EBNF grammar.)
Godot doesn’t reuse file handles
Section titled “Godot doesn’t reuse file handles”When reading from a pack file, Godot does not reuse a single open file handle for the pack. Instead, it will open a file handle on the pack, read the file, and then close the file handle. This is quite unusual, since it negates all I/O benefits of a virtual filesystem.
Since file paths in Godot can represent files in a pack file, as well as files in the game directory, the engine will also attempt two file opens when trying to read a file: one for the loose file and one for the main pack.
tinkering with making another @godotengine.org modloader and I've discovered something horrifying: why the hell does it open a separate handle for every file in the .pck??? what's even the point of using a virtual filesystem if you aren't going to reuse the handle lol
Replying to NotNite
I would expect N handles where N is the number of threads used to load assets into memory. I guess this was simpler to implement?
I am pretty sure resource loading only happens on a single thread based on what procmon says. can also see that it's attempting to load .gd and .import files from the host FS, which is probably yet another consequence of Godot mixing the VFS and host FS into the same APIs
