Why managing previous and predecessors per separate?

未关闭
#442 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
20/100
Issue 类型
重构
描述清晰度
需要澄清
活跃度
停滞
技术栈
python

调研方向

从 python/old_code/pywayang/src/pywy/orchestrator/operator.py 开始,重点查看 Operator 构造函数以及 getIterator、set_successor 和 set_predecessor 方法。在提出关系模型之前,追踪 previous、predecessor 和 successor 在旧版 PyWayang 代码的其他位置是如何使用的;完成这项工作需要一个达成一致的设计以及相应的 issue 范围。

由索引模型根据 Issue 内容生成。

描述

todo

Why managing previous and predecessors per separate?

https://github.com/apache/incubator-wayang/blob/c25c6561bf786d76d0dc717b28cf15885c269232/python/old_code/pywayang/src/pywy/orchestrator/operator.py#L69


#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import pickle
import cloudpickle
from pywy.config.config_reader import get_source_types
from pywy.config.config_reader import get_sink_types
from pywy.config.config_reader import get_boundary_types
import logging

pickle_protocol = pickle.HIGHEST_PROTOCOL


# Describes an Operation over an intermediate result
# Each operation could be processed by Python or Java platforms
class Operator:

    def __init__(
            self, operator_type=None, udf=None, previous=None,
            iterator=None, python_exec=False
    ):

        # Operator ID
        self.id = id(self)

        # Operator Type
        self.operator_type = operator_type

        # Set Boundaries
        if self.operator_type in get_boundary_types():
            self.boundary = True
        else:
            self.boundary = False

        # UDF Function
        self.udf = udf

        # Source types must come with an Iterator
        self.iterator = iterator
        if operator_type in get_source_types():
            if iterator is None:
                print("Source Operator Type without an Iterator")
                raise
            else:
                self.source = True
        else:
            self.source = False

        # Sink Operators
        if operator_type in get_sink_types():
            self.sink = True
        else:
            self.sink = False

        # TODO Why managing previous and predecessors per separate?
        self.previous = previous

        self.successor = []
        self.predecessor = []

        self.parameters = {}

        # Set predecessors and successors from previous
        if self.previous:
            for prev in self.previous:
                if prev is not None:
                    prev.set_successor(self)
                    self.set_predecessor(prev)

        self.python_exec = python_exec

        logging.info("Operator:" + str(self.getID()) + ", type:" + self.operator_type + ", PythonExecutable: " +
                     str(self.python_exec) +
                     ", is boundary: " + str(self.is_boundary()) + ", is source: " +
                     str(self.source) + ", is sink: " + str(self.sink))

    def getID(self):
        return self.id

    def is_source(self):
        return self.source

    def is_sink(self):
        return self.sink

    def is_boundary(self):
        return self.boundary

    def serialize_udf(self):
        self.udf = cloudpickle.dumps(self.udf)

    def getIterator(self):
        if self.is_source():
            return self.iterator
        # TODO this should iterate through previous REDESIGN
        return self.udf(self.previous[0].getIterator())

    def set_parameter(self, key, value):
        self.parameters[key] = value

    def set_successor(self, suc):
        if (not self.is_sink()) and self.successor.count(suc) == 0:
            self.successor.append(suc)

    def set_predecessor(self, suc):
        if self.predecessor.count(suc) == 0:
            self.predecessor.append(suc)

ea17eb1209cadc4bb2dbe95f752590de9723a51d

主要语言
Java
星标
284
派生
147
平均合并
2 天 13 小时
30 天内合并 PR
13

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

apache/wayang 的其他 Issue

查看 apache/wayang 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。