JSON in Delphi: Fast stream parser with System.JSON + UTF-8 pitfalls when handling special characters

„JSON in Delphi“ sounds like a solved problem: System.JSON is on board, REST-calls return text, done. In practice, the real errors occur where JSON is not available as a convenient string but as a stream: HTTP response stream, file stream, named pipe, message queue or a large BLOB from the database. Then three things coincide that are often underestimated in daily work: memory behavior, character encoding (especially UTF-8) and edge cases around special characters.

This article presents a clean, fast approach to parse JSON from a TStream without producing unnecessary copies — and above all without the typical UTF-8 traps where umlauts get „broken“ or parsers intermittently abort with cryptic messages. The focus is on operational and interface-stability implications: reproducible debugging, clear boundaries of the approach and criteria for when the effort is actually worthwhile.

Why streams behave differently for JSON parsing in Delphi

As long as a JSON document is small, the path „read stream into string, then parse“ is convenient. Beyond a certain payload size (typical: large lists, reports, sync data, log exports), however, this becomes expensive:

• Memory duplication: You read bytes into a buffer, convert to a Unicode string (Delphi string = UTF-16), and the parser creates additional internal structures. That can mean multiple copies briefly.
• GC/heap pressure: Many temporary strings and JSON values increase fragmentation and allocation overhead, especially in long-running processes (services, workers, import jobs).
• Error patterns become unclear: If incorrect encoding assumptions are made during reading, the JSON parser only sees „strange characters“ or unexpected control bytes.

It is important to keep a clear separation: JSON is formally Unicode; on the wire it is almost always UTF-8. Delphi however works internally with UTF-16. The transition from bytes (stream) to characters (string) is the point where special-character problems arise — not in the JSON itself.

System.JSON: What it does well — and where you need to be careful

System.JSON is the standard for DOM-based JSON in Delphi: you get an object model (TJSONObject, TJSONArray), can query values, iterate, serialize. This is robust for typical business integrations, but it has two consequences:

• It is not a true streaming parser: The object model is built in full. You may avoid reading into an extra string, but the DOM remains memory-intensive.
• The parser input is usually text: Depending on Delphi version and the API used, you quickly end up back at strings, including encoding conversion.

If your goal is a „fast stream parser“, in practice you usually mean one of two things: (1) no unnecessary copies and (2) as early fail-fast as possible for malformed payloads. Both are achievable with System.JSON, provided you control the byte-to-text stage.

UTF-8 pitfalls with special characters: the common causes

If umlauts (ä/ö/ü/ß) or other special characters look wrong in the output (ä, – etc.), this is almost always an encoding mismatch. In the Delphi environment these causes are particularly common:

1) ANSI fallback due to “convenient” helpers

Some reading paths silently assume the system ANSI encoding (code page of the Windows system) when no explicit encoding is provided. This only becomes noticeable when a payload contains more than ASCII. In test data that is often coincidentally “okay”; in production it fails with real names, places and free text.

2) BOM confusion (Byte Order Mark)

UTF-8 can start with a BOM (bytes EF BB BF). In the web context BOM is uncommon, but it can appear in files. Some readers detect the BOM and adjust the encoding, others do not or only in certain modes. If a BOM slips into the string as a normal character, you will often see an invisible “Zero Width No-Break Space” at the start or the JSON parser fails immediately on the first token.

3) Double conversion (UTF-8 is interpreted “again”)

The classic symptom (wrongly rendered umlaut) occurs when UTF-8 bytes are first correctly decoded to Unicode but later are misinterpreted again as ANSI/UTF-8 bytes (or vice versa). In Delphi this often happens when conversions between TBytes, RawByteString and string are ambiguous.

4) Truncation in the middle of a multibyte character

UTF-8 encodes special characters in 2–4 bytes. If you read in chunks (e.g. 8 KB) and the chunk boundary falls in the middle of a character, the decoder must buffer correctly. A naive approach that converts each chunk separately into a string and then concatenates them produces invalid sequences. This can appear as a “sporadic” error, depending on packet boundaries, proxy behavior or HTTP chunking.

5) Incorrect assumptions from HTTP headers

With REST the source is often Content-Type: application/json; charset=utf-8. Some servers, however, do not provide a charset, some provide incorrect information. If you rely blindly on the header, it can vary by backend version. For operations and support it is helpful to inspect the actual byte stream and log it in case of errors.

A clean approach: Stream → UTF-8-Decoder → JSON-Parser

The robust pipeline consists of three clear stages:

1. Read bytes from the stream (controlled, optionally with limit/timeout in the HTTP client).
2. Decode to Unicode with explicit UTF-8 (optionally tolerate BOM).
3. Parse with System.JSON into an object model or into targeted extraction.

The most important lever is stage 2: you do not want „Default Encoding“ to decide anywhere. In Delphi this means: set TEncoding.UTF8 explicitly and do not rely on implicit conversions.

What „quick“ specifically means here

With System.JSON you won’t „optimize away“ the DOM. But you can avoid:

• an additional copy of the entire payload as an intermediate string when you only need a few values internally anyway (in that case another parser is usually more appropriate; more on that later),
• multiple re-encodings,
• and you can read very large payloads in a controlled way (with size limits and a clear error message) instead of ending up with Out-of-Memory or Access Violations.

The concrete edge case: special characters corrupted, but only intermittently

One practical edge case is particularly insidious: the payload is valid UTF-8 JSON in principle, but you read it in chunks and convert each chunk to a string. As long as only ASCII appears, you notice nothing. As soon as an umlaut falls exactly on a chunk boundary, invalid UTF-8 sequences are created. Result: either garbled characters or a parse error at a location that does not match the actual content.

How to recognize it:

• Parse errors occur „randomly“ for large responses, not for small ones.
• The same request sometimes succeeds and sometimes fails (depending on chunking/transport).
• A hex dump of the bytes shows valid UTF-8, but your logged string contains Replacement Characters (�) or classic mojibake.

The solution is not to „read more lines“ or use „larger buffers“, but to use a decoder that correctly buffers multibyte sequences across chunk boundaries. That’s exactly where TStreamReader used with a UTF-8 encoding can be helpful — if you initialize it correctly.

Practical guide: Reproducibly checking UTF-8 in Delphi

Before you touch the parser, you need a debugging setup that reveals the actual byte sequence. For support and operations this is invaluable, because you can later state clearly whether the remote endpoint is sending incorrect data or your pipeline decodes it incorrectly.

1) Check the first bytes (BOM, JSON start)

If the JSON comes with a BOM, you see EF BB BF at the start of the stream. Immediately after should typically follow „{“ or „[„. If you already see „“ in the string, the BOM was not treated as a BOM but decoded as text.

2) Log raw bytes in hex — but with limits

Do not log complete payloads in production (data protection, cost, log volume). Proven approaches are:

• prefix (e.g. first 256 or 1024 bytes),
• suffix (last 256 bytes),
• and a hash (SHA-256) for correlation when you need to compare payloads.

With that you can often classify special-character problems within minutes: Is the byte sequence for „ä“ correct (C3 A4)? Is there truncation? Does an unexpected 0x00 (null byte) appear, e.g. due to an incorrect assumption of UTF-16?

3) Log Content-Type and charset

For HTTP/REST: log the Content-Type and the declared charset. If the bytes are clearly UTF-8 but the charset claims otherwise, you should not blindly follow it in the client. For JSON, UTF-8 is the de facto standard. When in doubt: byte analysis wins over headers.

Fast stream parser with System.JSON: design without unnecessary copies

A practical pattern is: read from the stream into a byte buffer, construct a string from it exactly once using UTF-8, and pass that string to the JSON parser. This is not „streaming“ in the SAX sense, but it is a controlled, performant pipeline without unexpected encoding shifts.

Important for the architecture: design the function so it makes the decision centrally in one place:

• Which encoding rule applies (typically UTF-8, BOM optional)?
• What is the maximum allowed payload size (DoS protection, operational limit)?
• What do error messages look like (with context, but without leaking data)?

Read strategy: bounded buffering instead of „StreamToString“ without a limit

If you accept JSON from external sources (partners, mobile clients, third parties), a size limit is mandatory. Without a limit, a single unlucky request is enough to put a service under memory pressure. Practically this means: count the total bytes while reading and abort once a threshold is reached—with a clear exception that is understandable in monitoring.

Why I often expect UTF-8 „without BOM“, but would tolerate a BOM

BOMs are rare in REST payloads. They occur more often in files (exports, manual editing). For robust import paths, it’s sensible to tolerate a BOM but make it visible in the logs, because it can indicate „file-world“ rather than „API-world“.

The silent killers: TStreamReader defaults and mixed use of text readers

TStreamReader is convenient, but you must have two things under control:

• Set encoding explicitly: Don’t hope it will „detect“ it.
• Understand buffering: The reader buffers internally. If you read the same stream elsewhere later, the stream position matters. This sounds trivial but becomes an error source quickly in larger import pipelines.

Mixed operation is particularly unpleasant: first read part as bytes (e.g. for logging or magic bytes), then continue with TStreamReader. If you don’t rewind cleanly or initialize the reader at the correct position, you may read from byte 257 instead of 0. The JSON parser will then report „Invalid character at position …“, even though the payload itself is correct.

When special characters remain „broken“ despite UTF-8: escaping vs. real Unicode

JSON can contain special characters in two ways:

• As real UTF-8 characters (e.g. „München“ as bytes C3 BC …).
• As an escape sequence (e.g. „Mu00fcnchen“).

Both are valid. For practical purposes: escape sequences avoid many transport issues, but they only superficially mask encoding errors. If your system interprets bytes incorrectly anywhere, that is an operational risk, not just a cosmetic bug. In addition, escape sequences can confuse logging/monitoring when teams expect „readable text“.

System.JSON will ultimately give you normal Delphi strings (UTF-16) in both cases, provided the path to that point was correct.

Assess performance realistically: DOM costs, large arrays and selectivity

The biggest performance lever is often not „make the parser faster“ but parse less. With System.JSON this is difficult because you get the DOM. Three typical situations:

• Large arrays (10,000+ elements): Building the DOM costs time and RAM. If you only need 2 fields per element, a streaming-capable parser (SAX/tokenizer) is often more appropriate. System.JSON is not designed for that.
• Single objects with many fields: If you only need a few fields, you can still use the DOM, but avoid traversing multiple times. Retrieve values once and map them into your structures.
• Multiple large payloads in sequence: In import jobs or sync workers it pays off to encapsulate parsing into a clear step and release all references after each document so the memory manager can clean up. It’s trivial, but in services this is often done „on the side“.

A „fast stream parser“ with System.JSON is therefore often a good compromise: read efficiently and correctly, use the DOM consciously, define boundaries. If you need true streaming semantics (e.g. processing array elements one by one without retaining everything), System.JSON is not the right foundation.

Operational robustness: error patterns and how to make them immediately actionable

JSON parse errors in logs are often not helpful because they only report a position. For operations and support you need context:

• Byte position vs. character position: With UTF-8 these are not identical. If the parser reports a character position, the byte position may differ. For byte dumps the byte position is decisive.
• Snippet around the error location: Log a small window around the position on error (e.g. 40 characters before/after), but only if no sensitive data is included. Alternatively: log only hexadecimal.
• Correlation: Request ID, endpoint, partner ID, payload hash. Otherwise you’ll never find „that one error“ again.

The goal is that after a production incident you can answer within minutes: „encoding misinterpreted“, „payload truncated“, „server returns invalid JSON“ or „we have a mapping problem“.

Avoid UTF-8 pitfalls deliberately: checklist

• Always specify the encoding: When reading from streams and when writing to logs/files, do not rely on defaults.
• No chunk-to-string conversion: If you read chunked, collect bytes or use a decoder that buffers multibyte sequences.
• BOM tolerant but visible: Accept it, but be able to detect it during debugging.
• Set limits: Max payload size, max object/array depth (where you can control it), timeouts in the HTTP client.

When is the effort for a stream parser really worthwhile?

You don’t have to optimize every JSON occurrence. The approach is typically worthwhile if at least one of the following applies:

• Large payloads (several MB) occur regularly or can occur.
• Long-running processes (service, worker) process many payloads and you observe memory spikes or fragmentation.
• Interop with heterogeneous systems: multiple partners, different platforms, occasionally faulty encodings.
• Incident history: there have already been garbled umlauts, sporadic parse errors, or hard-to-reproduce import failures.

If your payloads are small and come from a controlled source, a simple approach is often sufficient — but even then: explicitly setting UTF-8 costs almost nothing and prevents later surprises.

Distinction: when you need true streaming

System.JSON is DOM-oriented. If you really want to process data “on the fly”, e.g. a large array element by element without holding it completely, you need a different parser approach (tokenizer/SAX). This is not a value judgment but an architectural decision:

• DOM (System.JSON): convenient, suitable for typical business objects, but memory-intensive.
• Streaming/SAX: lower memory footprint, suitable for very large data, but requires more implementation effort and more careful error handling.

A reasonable compromise is often: implement stream handling, encoding and limits cleanly, and only then decide whether DOM still fits. In many projects that alone significantly stabilizes operation.

Conclusion: JSON in Delphi becomes reliable when you treat encoding and streams as a separate layer

Most problems around “JSON in Delphi” are not caused by the JSON parser itself, but by the inconspicuous stretch before it: bytes from a stream are turned into text. If you explicitly handle UTF-8 there, detect BOM cases, do not ignore chunk boundaries and set clear size limits, the typical special-character errors disappear — and sporadic parse problems become reproducible.

System.JSON remains a pragmatic standard: not the fastest streaming parser, but solid if you control the input and consciously accept the DOM costs. If you like, we can walk through your specific import/REST path together and identify the point where encoding or chunking fails: get in touch.

JSON stream parsers are also important for this topic. The article places these aspects in context and shows what matters in practice.

Discuss a project or modernization initiative with Net-Base.