Disabled CUSTOM types still serialize through OJSONWriter

Open Beginner friendly
#11,010 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java
Domain
backend, databases

Research direction

Read core/src/main/java/com/orientechnologies/orient/core/serialization/serializer/OJSONWriter.java around line 173 and compare its CUSTOM handling with the other serializers’ checks for DB_CUSTOM_SUPPORT. Start with the existing OJsonWithCustom.testCustomFieldDisabled test and add coverage for ODocument.toJSON() with custom support disabled. Done when the write path rejects or skips the CUSTOM value and no longer emits its u marker and Base64 payload.

Written by the indexing model from the issue text.

Description

OrientDB Version: 3.2.55 (build 5a42e4f47183a1db1fa00186bd817dda57e24eeb)
Java Version: OpenJDK 1.8.0_492 (jdk8u492-b09)
OS: Linux x86_64 (official orientdb:latest image, Docker)

Expected behavior

When DB_CUSTOM_SUPPORT / db.custom.support is false, CUSTOM types should be disabled consistently. The JSON write path (ODocument.toJSON() / OJSONWriter) should not emit a CUSTOM @fieldTypes marker (u) and a Java-serialized Base64 payload; it should reject the write with the same kind of error as the other serializers, or skip the value.

Actual behavior

With custom support disabled, ODocument.toJSON() still produces the custom representation:

{"@type":"d","@version":0,"@fieldTypes":"customField=u","customField":"rO0..."}

The PoC reports HAS_FIELD_TYPES=true, HAS_CUSTOM_TYPE_MARKER=true, HAS_JAVA_SERIALIZED_PAYLOAD=true.

Deserializing that same JSON while support is disabled then fails with:

ODatabaseException: OType CUSTOM used with serializable types, is not enabled...

So the write path produces data that the same configuration refuses to read.

Steps to reproduce

This is a Java API-level reproduction using the runtime jars from the official image.

  1. Start the container (the image provides javac and /orientdb/lib/*):
docker run -d --name orient-custom -p 127.0.0.1:32480:2480 \
  -e ORIENTDB_ROOT_PASSWORD=auditpass orientdb:latest
  1. Compile and run a small PoC inside the container:
// outline
OGlobalConfiguration.DB_CUSTOM_SUPPORT.setValue(true);
ODocument doc = new ODocument("CustomDoc");
doc.field("customField", new MySerializableValue(...)); // OType.CUSTOM
OGlobalConfiguration.DB_CUSTOM_SUPPORT.setValue(false);
String json = doc.toJSON();
System.out.println(json);
  1. Observe the output contains "@fieldTypes":"customField=u" and a Base64 Java-serialized customField value.

  2. Feed the same JSON back to the deserializer while support is disabled; it throws OType CUSTOM used with serializable types, is not enabled....

The PoC creates the field while support is enabled and then flips it off, so the failure is isolated to the write path.

Additional context

  • core/src/main/java/com/orientechnologies/orient/core/serialization/serializer/OJSONWriter.java:173 — the valueType == OType.CUSTOM branch creates an ObjectOutputStream, serializes the object and writes Base64, but never reads or checks DB_CUSTOM_SUPPORT.
  • OGlobalConfiguration.DB_CUSTOM_SUPPORT is documented as “Enables or disables use of custom types”.
  • ODocument.field(...) / ODocument.setFieldType(...) and the JSON/binary/network serializers already check the flag and throw when CUSTOM is used while disabled.
  • The existing OJsonWithCustom.testCustomFieldDisabled test covers deserialization, but it does not assert that toJSON() also refuses to write custom content.

I re-checked current develop on 2026-11-17: OJSONWriter.java still has the CUSTOM branch without a DB_CUSTOM_SUPPORT check.

Suggested fix: check DB_CUSTOM_SUPPORT in the OJSONWriter CUSTOM branch and throw the same ODatabaseException (or skip the value with a diagnostic) used by the other serializers; add a test that calls toJSON() with the flag off and asserts it throws/skips instead of emitting u + Base64.

Scope note: this is a local Java API reproduction, not an HTTP/SQL surface. The PoC uses the runtime jars from the 3.2.55 image.

Dominant language
Java
Stars
5k
Forks
871
Avg merge
17h 38m
Merged PRs (30d)
40

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from orientechnologies/orientdb

All issues in orientechnologies/orientdb

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.