[Improve] Gremlin query ids batch query to backend such as HBase/HStore

Open
#2,674 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
java
Domain
backend, databases

Research direction

Start with GraphTransaction#queryVerticesByIds and trace how IdQuery is sent through queryVerticesFromBackend for HBase/HStore. Compare the current per-ID query.query(id) flow with the proposed batched query.query(ids) approach. Done means Gremlin queries such as g.V('id1','id2','id3') batch backend lookups while preserving the local transaction and result-mapping behavior shown.

Written by the indexing model from the issue text.

Description

bug gremlin hbase improvement
Bug Type (问题类型)

None

Before submit
  • 我已经确认现有的 IssuesFAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)
Environment (环境信息)
  • Server Version: master
  • Backend: HBase/HStore
  • OS: CentOS 7.x
Expected & Actual behavior (期望与实际表现)
Motivation
1. 在时机使用中,发现批量点查性能差,无法满足业务需求
2. 提升批量查询性能
Example
1. 查询语句 g.V('id1','id2','id3')

当前版本问题 时间复杂度为O(n),
当使用RPC查询后端,性能很差,常规优化手法是批量查询下发存储层

Optimization plan

此处直接调用query.query(ids)

TODO: 待提交PR

GraphTransaction#queryVerticesByIds

protected Iterator<Vertex> queryVerticesByIds(Object[] vertexIds,
                                                  boolean adjacentVertex,
                                                  boolean checkMustExist,
                                                  HugeType type) {
        Query.checkForceCapacity(vertexIds.length);

        // NOTE: allowed duplicated vertices if query by duplicated ids
        List<Id> ids = InsertionOrderUtil.newList();
        Map<Id, HugeVertex> vertices = new HashMap<>(vertexIds.length);

        IdQuery query = new IdQuery(type);
        for (Object vertexId : vertexIds) {
            HugeVertex vertex;
            Id id = HugeVertex.getIdValue(vertexId);
            if (id == null || this.removedVertices.containsKey(id)) {
                // The record has been deleted
                continue;
            } else if ((vertex = this.addedVertices.get(id)) != null ||
                       (vertex = this.updatedVertices.get(id)) != null) {
                if (vertex.expired()) {
                    continue;
                }
                // Found from local tx
                vertices.put(vertex.id(), vertex);
            } else {
                // Prepare to query from backend store
                query.query(id);
            }
            ids.add(id);
        }

        if (!query.empty()) {
            // Query from backend store
            query.mustSortByInput(false);
            Iterator<HugeVertex> it = this.queryVerticesFromBackend(query);
            QueryResults.fillMap(it, vertices);
        }

        return new MapperIterator<>(ids.iterator(), id -> {
            HugeVertex vertex = vertices.get(id);
            if (vertex == null) {
                if (checkMustExist) {
                    throw new NotFoundException(
                            "Vertex '%s' does not exist", id);
                } else if (adjacentVertex) {
                    assert !checkMustExist;
                    // Return undefined if adjacentVertex but !checkMustExist
                    vertex = HugeVertex.undefined(this.graph(), id);
                } else {
                    // Return null
                    assert vertex == null;
                }
            }
            return vertex;
        });
    }
Dominant language
Java
Stars
3.2k
Forks
637
Avg merge
3d 18h
Merged PRs (30d)
23

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/hugegraph

All issues in apache/hugegraph

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.