Can `ResumableParser` return whether the last `partial_value` container is completed or not?

Open
#1,041 7 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
ruby
Domain
backend

Research direction

Start by reproducing the two JSON::ResumableParser chunk examples in the issue and inspect how partial_value is populated after parse. Trace the parser state for a completed versus incomplete last record; done means callers can reliably distinguish those cases, including the workaround scenarios described.

Written by the indexing model from the issue text.

Description

This is not a critical feature request.

I'm trying to migrate to json gem from json-stream gem in https://github.com/groonga/groonga-command-parser/ by JSON::ResumableParser.

It parses records to be loaded to Groonga (a full text search engine with column store). The records use the following format:

[
{"column_name1": "record1's column_value1", "column_name2": 2, ...},
{"column_name1": "record2's column_value1", "column_name2": 2, ...},
...
]

groonga-command-parser wants to process a record as soon as it's completed. For example, it wants to process the first record when the following chunk is processed:

[
{"column_name1": "record1's column_value1", "column_name2": 2, ...},

We can do it by the following script:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1"},
CHUNK
parser.parse
p parser.partial_value[0] # => {"column1" => "value1"}

But it doesn't work with the following chunk:

[
{"column_name1": "record1's column_value1",

We can't know whether the record is completed or not:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1",
CHUNK
parser.parse
p parser.partial_value[0] # => {"column1" => "value1"} # This is not completed yet

groonga-command-parser has a workaround for it. It ignores the last record:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1"},
CHUNK
parser.parse
p parser.partial_value[0..-2] # => []

parser << <<-CHUNK
{"column1": "value2"},
CHUNK
parser.parse
p parser.partial_value[0..-2] # => [{"column1" => "value1"}] # 1st record

If we can detect whether a record in this example is completed or not, it's useful. But this is not a critical feature request because there is a workaround for this case.

Dominant language
Ruby
Stars
785
Forks
390
Avg merge
7h 14m
Merged PRs (30d)
15

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ruby/json

All issues in ruby/json

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.