Streaming Large JSON Files in Python

The working set matters more than file size alone

A multi-gigabyte JSON file can exceed memory even when the file itself appears smaller than the machine's available RAM. Decoding creates text, containers, keys, and values, often while the original bytes remain live. The expansion factor depends on the data and runtime, so multiplying file size by a universal constant is unreliable. Measure the actual representation and object lifetime.

Whole-document loading is appropriate for bounded small inputs, but it is a poor default for large arrays when records can be processed independently. An incremental parser lets the application act on one selected value at a time. The memory budget then depends on parser buffers, the largest selected record, and downstream retained state rather than the total number of records.

That guarantee is conditional. A single enormous record can still be enormous, and a consumer that appends every yielded object to a list recreates whole-document retention. Review the entire pipeline before describing it as streaming. If you control the producer, the NDJSON guide offers simpler record framing. If you must consume an existing large array, ijson can avoid loading the entire array at once.

Choose the right parser interface and prefix

ijson exposes object-oriented iteration as well as lower-level parsing events. Selecting items under an array prefix is convenient when each record is reasonably bounded. Event processing is useful when only a few fields are needed or individual objects are too large to materialize comfortably. The choice determines how much application structure is built.

Verify the prefix against a small representative document before running a long import. A mistaken prefix can yield no records without matching the intended data path, and an incorrect assumption about the root shape can make a job appear successfully empty. Assert expected counts or structural markers when the source contract provides them.

Open files in binary mode and keep encoding policy explicit. Backend selection and numeric options can affect performance and behavior, so record the chosen backend and dependency version with benchmark results. Do not enable a faster numeric mode without checking precision requirements. A decimal total that becomes a binary float can change financial results even when the parser is faster and uses less memory.

A self-contained streaming and memory experiment

The example writes a modest synthetic array incrementally, reads records through ijson, and accumulates only a count and integer total. It uses a temporary directory so running it does not overwrite application data. Install ijson 3.x and save the program as stream_array.py. The default record count is small enough for a local smoke test; increase it deliberately within your test environment's disk budget.

The fixture generator also avoids building the entire array in memory. Otherwise the benchmark setup itself could fail before testing the streaming reader. The parser selects each array item, and the loop discards it after updating bounded aggregates. Integer minor units keep this example's arithmetic exact without requiring a custom decimal output serializer.

The reported tracemalloc peak describes traced Python allocations, not complete process resident memory. Native parser allocations and allocator behavior can differ. Use an operating-system process metric for the full memory envelope and compare several input sizes in separate processes. A flat traced peak is useful evidence but not proof that every byte of the process is bounded identically.

import json
import tempfile
import tracemalloc
from pathlib import Path
import ijson

with tempfile.TemporaryDirectory() as directory:
    path = Path(directory) / "records.json"
    count = 20_000
    with path.open("w", encoding="utf-8") as output:
        output.write("[")
        for index in range(count):
            if index:
                output.write(",")
            json.dump({"id": index, "amountMinor": 125}, output)
        output.write("]")

    tracemalloc.start()
    seen = 0
    total = 0
    with path.open("rb") as source:
        for record in ijson.items(source, "item"):
            seen += 1
            total += record["amountMinor"]
    current, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    assert seen == count
    assert total == count * 125
    print({"records": seen, "totalMinor": total,
           "pythonPeakBytes": peak})

Generators do not guarantee bounded retention

A generator delays production, but its consumer decides whether values remain live. Calling list on the generator, submitting every record to an unbounded executor, or storing all errors in memory defeats the intended bound. Inspect accumulators, retry lists, caches, and closures as carefully as the parser itself.

Use a bounded batch with both count and byte constraints when writing to a database or API. Clear completed batches and avoid keeping references through metrics or debugging state. If the sink is asynchronous, cap active tasks and await completion before reading too far ahead. Backpressure should stop source consumption when the sink cannot keep up.

Aggregation can also grow without bound. A sum is constant state, while a dictionary keyed by every distinct customer grows with cardinality. For large group-by operations, partition data, spill to disk, or use a database designed for the workload. The PostgreSQL guide discusses moving selective and aggregate work into an indexed store when keeping a Python object for every group is the wrong architecture.

Precision and malformed data need explicit policies

Choose numeric behavior according to the domain. Decimal-preserving parsing is useful when fractional fidelity matters, while floating point may be appropriate for approximate measurements. Integers can still have application bounds even if the runtime supports large values. Validate types and ranges before arithmetic and before binding values into a destination system with narrower types.

Handle malformed input consistently. A broken whole-array document may not offer a safe arbitrary resume point, unlike a well-framed record stream with known boundaries. Do not search for the next opening brace and assume parsing can resume correctly; braces can occur inside strings and nested objects. Fail the job or use a documented recovery format rather than inventing structural repair.

Strict transport decoding remains important. If bytes are replaced or misdecoded before parsing, valid-looking records can carry corrupted identifiers. The Unicode guide provides fixtures for these boundaries. Keep error reports bounded and identify the source and processing stage without retaining the full sensitive document in an exception log.

Profile memory in separate controlled runs

Run whole-document and incremental variants in separate processes to avoid allocator history contaminating peak measurements. Record input bytes, record count, largest record, runtime version, parser backend, and elapsed time. Compare several scales so the growth trend is visible. A single input cannot establish whether memory grows with total records or only with record size.

Use resident memory from the operating system alongside Python allocation tracing. Platform tools differ in units and semantics, so label them carefully. Peak resident memory includes more than live Python objects, while traced allocations omit some native behavior. Neither metric should be renamed simply memory without explaining what it measures.

Warm filesystem caches can affect throughput substantially. Separate cold-read and warm-read experiments when disk performance matters, and avoid claiming parser improvements from changes in cache state. Measure CPU and wall time independently where useful. Keep logging outside the hot loop or sample it sparsely, because printing every record can dominate the experiment and conceal parsing costs.

Design restart and sink semantics together

Streaming a large array does not automatically provide a cheap durable checkpoint. The parser's buffered position may not correspond to a safe restart boundary for an individual record. If restartability is essential, consider converting the source once into a record-framed staging format or partitioning it into independently valid documents.

Make sink writes idempotent when replay is possible. A stable source record identifier can support upserts or deduplication, but the database operation must actually enforce that identity. Record job version and source identity so a restart cannot accidentally combine different input files. Do not mark a record complete before its durable effect succeeds.

For long imports, expose progress using counts and source metadata rather than assuming byte position maps linearly to work. Records vary in size and processing cost. Bound dead-letter storage and stop when a high failure rate indicates a systematic contract mismatch. A job that keeps streaming invalid data efficiently can still waste substantial capacity and produce an unmanageable error backlog.

Choose an operating envelope and verify it

Set limits for source size, individual record size where enforceable, nesting, batch memory, concurrent sink operations, and job duration. Document which limits are enforced by the parser and which are enforced by the surrounding process or infrastructure. If a required structural limit cannot be enforced by the selected interface, acknowledge that and choose a different parser mode or isolation strategy.

Test an oversized record, a slow sink, malformed input near the end, and a process interruption. Verify that temporary files are cleaned up and that a failed job does not publish a partial result as complete. For exports, write to a staging destination and commit or rename only when the full operation succeeds under the storage system's semantics.

Finally, keep the implementation simple enough to audit. A straightforward iterator with a bounded batch is often easier to operate than a highly concurrent pipeline whose queues obscure memory ownership. Optimize only after measuring the bottleneck. Streaming is most valuable when it changes the working-set model predictably, not when it merely replaces one parser function name with another.

Use event-level parsing for unusually large records

Item iteration is convenient, but it still constructs each selected item. If one item contains a massive nested list and the job needs only a scalar field, consider the lower-level event interface. Maintain only the state required for the calculation and ignore irrelevant subtrees under a documented policy.

Event processing increases application complexity. You must distinguish container boundaries, repeated keys, and the selected path correctly. Build a small fixture with nested arrays and similarly named fields at different levels so a broad suffix match cannot accidentally aggregate unrelated values. Prefer an exact structural condition where the contract allows it.

Do not describe event processing as automatically constant memory. A parser may buffer a large scalar, and your own state can still grow with distinct keys or collected values. Define the largest accepted scalar and the aggregation cardinality. If the interface cannot enforce a needed limit before allocation, use process isolation or a more suitable ingestion format.

Keep correctness tests identical between the item-based and event-based implementations. Compare counts, totals, missing-field handling, and malformed-input rejection before measuring performance. The lower-level version is worthwhile only if it changes a measured resource constraint enough to justify the additional code.

Plan multi-gigabyte jobs as restartable data products

A large import should have an immutable source identity, processing version, output destination, and completion marker. Do not expose partially written output as a finished dataset. Write to a staging location and publish the result only after validation and sink completion satisfy the job's contract.

Estimate disk and memory headroom before starting. Temporary decompressed files, output batches, and error records can require more storage than the original input. Streaming reduces one working set but may introduce staging storage. Include these resources in admission control so several simultaneous jobs do not exhaust the same volume.

For partitioned processing, choose boundaries that preserve independent valid documents or record frames. Splitting a JSON array at arbitrary byte positions is not safe. If a preprocessing pass creates partitions, validate and record their identities so downstream workers can retry them independently without guessing where a record begins.

Checkpoint durable progress at a level the source format supports. A job that cannot safely resume within one large array can still restart from a partition or use idempotent destination writes. Be honest about the replay cost and test it. A theoretical checkpoint that cannot reconstruct parser state is worse than a clear restart policy.

Finally, report progress and failure categories in bounded metadata. Include processed record count, completed partitions, and sink status. Avoid retaining every successful record identifier in memory merely to display progress. The same discipline that keeps parsing bounded should extend to the job manager, status endpoint, and retry bookkeeping.

Record the selected ijson backend in job diagnostics because available native extensions can differ between development and deployment. The same Python entrypoint may therefore use a different implementation. Reproduce performance findings on the target environment before changing batch sizes or promising a particular import completion time.

Engineering Comparison

The experiment reports traced Python allocations. Measure process RSS separately and do not interpret one successful file size as a universal memory guarantee.