[Improve] Gremlin query ids batch query to backend such as HBase/HStore
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
Research direction
Start with GraphTransaction#queryVerticesByIds and trace how IdQuery is sent through queryVerticesFromBackend for HBase/HStore. Compare the current per-ID query.query(id) flow with the proposed batched query.query(ids) approach. Done means Gremlin queries such as g.V('id1','id2','id3') batch backend lookups while preserving the local transaction and result-mapping behavior shown.
Written by the indexing model from the issue text.
Description
Bug Type (问题类型)
None
Before submit
- 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)
Environment (环境信息)
- Server Version: master
- Backend: HBase/HStore
- OS: CentOS 7.x
Expected & Actual behavior (期望与实际表现)
Motivation
1. 在时机使用中,发现批量点查性能差,无法满足业务需求
2. 提升批量查询性能
Example
1. 查询语句 g.V('id1','id2','id3')
当前版本问题 时间复杂度为O(n),
当使用RPC查询后端,性能很差,常规优化手法是批量查询下发存储层
Optimization plan
此处直接调用query.query(ids)
TODO: 待提交PR
GraphTransaction#queryVerticesByIds
protected Iterator<Vertex> queryVerticesByIds(Object[] vertexIds,
boolean adjacentVertex,
boolean checkMustExist,
HugeType type) {
Query.checkForceCapacity(vertexIds.length);
// NOTE: allowed duplicated vertices if query by duplicated ids
List<Id> ids = InsertionOrderUtil.newList();
Map<Id, HugeVertex> vertices = new HashMap<>(vertexIds.length);
IdQuery query = new IdQuery(type);
for (Object vertexId : vertexIds) {
HugeVertex vertex;
Id id = HugeVertex.getIdValue(vertexId);
if (id == null || this.removedVertices.containsKey(id)) {
// The record has been deleted
continue;
} else if ((vertex = this.addedVertices.get(id)) != null ||
(vertex = this.updatedVertices.get(id)) != null) {
if (vertex.expired()) {
continue;
}
// Found from local tx
vertices.put(vertex.id(), vertex);
} else {
// Prepare to query from backend store
query.query(id);
}
ids.add(id);
}
if (!query.empty()) {
// Query from backend store
query.mustSortByInput(false);
Iterator<HugeVertex> it = this.queryVerticesFromBackend(query);
QueryResults.fillMap(it, vertices);
}
return new MapperIterator<>(ids.iterator(), id -> {
HugeVertex vertex = vertices.get(id);
if (vertex == null) {
if (checkMustExist) {
throw new NotFoundException(
"Vertex '%s' does not exist", id);
} else if (adjacentVertex) {
assert !checkMustExist;
// Return undefined if adjacentVertex but !checkMustExist
vertex = HugeVertex.undefined(this.graph(), id);
} else {
// Return null
assert vertex == null;
}
}
return vertex;
});
}
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 637
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 23
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/hugegraph
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
Difficulty 3/5 1-2 days Newbie friendliness 64/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 28/100
All issues in apache/hugegraph
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
opensearch-project/k-NN#3597 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100