Support simpler and easier streaming parsing for nested objects

Open
  • Is something wrong?
#6,330 7 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
java, json
Domain
backend

Research direction

This is a design-level feature request for lazy/streaming deserialization of nested collections in jackson-databind. Start by reading the linked issue #1304 and the maintainer discussion in the comments, then explore how ObjectReader and the deserializer for collection types currently bind the whole payload. Several alternative APIs (Iterable/Stream wrappers, annotations, callbacks) are proposed without a decision, so 'done' first means an agreed design with maintainers, not a patch.

Written by the indexing model from the issue text.

Description

to-evaluate
Is your feature request related to a problem? Please describe.

When integrating with some third-party APIs, there is sometimes a very large amount of data, and the relatively large data is inside an object, so there is no simple and easy way to parse it in a streaming manner.

{
    "code": 200,
    "message": "xxx",
    "data": [
        {
            "id": 1,
            "name": "xxx",
            "age": 18
        },
        {
            "id": 2,
            "name": "xxx",
            "age": 18
        }
        // ... a lot json data
    ]
}

But I have no way to ask the API provider to change the structure, so I would like to support this kind of nested streaming parsing.

Describe the solution you'd like

I want to use something like the following:

import lombok.Data;

@Data
public class ListUserInfoResponse {

    private Integer code;

    private String message;

    private Iterable<Data> data;

    @lombok.Data
    public static class Data {

        private Integer id;

        private String name;

        private Integer age;
        
    }

}

ListUserInfoResponse can be loaded normally, but the data inside it is loaded in a streaming and lazy manner.

It does not have to use Iterable; it could also be wrapped in Jackson's own data class (preferably one that implements Iterable or Stream).

Alternatively, directly supporting Iterable or Stream is also possible, but it may be difficult to adapt, which is why Jackson's own data class may be needed.

Or it could be done by using an annotation on this class field.

However, this approach may not be friendly to things like InputStream that need to be closed, since it is lazy-loaded and may be closed midway.

So my idea is to first read the entire InputStream completely, and automatically convert it to ListUserInfoResponse.

When a property of a specific type or marked with a specific annotation is encountered, convert it to an Iterable or Stream implemented by Jackson's own implementation class, which stores the partial string read from the InputStream, that is, the string for this property.

Only when this property is iterated over does parsing begin according to the type. Personally, I feel Iterable is indeed better; for iterable data of unknown size, it is indeed more appropriate than data types like List whose size is known.

Alternatively, use a callback function for things like InputStream that need to be closed, and automatically close it after the callback finishes (but this seems not very friendly to imperative blocking operations).

Usage example

No response

Additional context

Personally, I think that although the entire string also occupies a large amount of memory, and directly using low-level operations to read may occupy less memory, I think this solution is better in terms of convenience and ease of use, and it also saves more memory than the approach without streaming lazy loading. It is a relatively good compromise.

For small amounts of data, just load it directly; for very large datasets, use low-level operations to read; if it is relatively large and you also want to reduce memory usage, I think this solution is better.

https://github.com/FasterXML/jackson-databind/issues/1304
https://juneau.staged.apache.org/docs/topics/LargeDatasetStreaming/#in-memory-listbeanchannel

Dominant language
Java
Stars
3.8k
Forks
1.6k
Avg merge
1d 18h
Merged PRs (30d)
59

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from FasterXML/jackson-databind

All issues in FasterXML/jackson-databind

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.