Unstructured-IO/unstructured

feat/clean_newline

Offen

#2.513 geöffnet am 06.02.2024

 (4 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)HTML (1.232 Forks)batch import
enhancementgood first issue

Repository-Metriken

Stars
 (14.711 Sterne)
PR-Merge-Metriken
 (Durchschn. Merge 3T 1h) (9 gemergte PRs in 30 T)

Beschreibung

Is your feature request related to a problem? Please describe. Since often words that continue on the following line are described as a character followed by a dash and 1+ whitespaces it would be useful to have a function clean_newline that concatenates the text on newline.

def clean_newline(text: str, pattern: str = r"(\w+)-\s+(\w+)" ) -> str:
    """
    The `clean_newline` function removes the hyphen and whitespace between two words in a given text.
    
    :param text: A string that contains the text to be cleaned
    :type text: str
    :return: a modified version of the input text where any occurrence of a word followed by a hyphen
    and whitespace, followed by another word, is replaced with just the two words concatenated together.
    """
    return re.sub(pattern, r'\1\2', text)

Contributor Guide