Unstructured-IO/unstructured

feat/clean_newline

开放

#2,513 创建于 2024年2月6日

 (4 条评论) (0 个反应) (0 位负责人)HTML (1,232 个派生)batch import
enhancementgood first issue

仓库指标

星标
 (14,711 个星标)
PR 合并指标
 (平均合并 3天 1小时) (30 天内合并 9 个 PR)

描述

Is your feature request related to a problem? Please describe. Since often words that continue on the following line are described as a character followed by a dash and 1+ whitespaces it would be useful to have a function clean_newline that concatenates the text on newline.

def clean_newline(text: str, pattern: str = r"(\w+)-\s+(\w+)" ) -> str:
    """
    The `clean_newline` function removes the hyphen and whitespace between two words in a given text.
    
    :param text: A string that contains the text to be cleaned
    :type text: str
    :return: a modified version of the input text where any occurrence of a word followed by a hyphen
    and whitespace, followed by another word, is replaced with just the two words concatenated together.
    """
    return re.sub(pattern, r'\1\2', text)

贡献者指南