facebook/duckling

Arabic regex match some parts of words that mustn't be matched

Open

#131 opened on Dec 24, 2017

View on GitHub
 (4 comments) (5 reactions) (0 assignees)Haskell (4,282 stars) (737 forks)batch import
bughelp wanted

Description

In some Arabic dimensions, the regex doesn't match whole words only, instead it sometimes match some part of the word that mustn't be matched at all ! which makes serious buggy outputs. I noticed this problem in the following dimensions: 1- ordinal 2- number 3- time 4- duration

for example: if the input text was: "وأحدث ذلك مشكلة كبيرة" which in English means: "And that made a big problem"

Duckling match the word "احد" (which means "Sunday") from its built-in regex, with the word "وأحدث" (which means "made") from the input text ! (see the following photo for run time) time bug

this was in "time" dimension.

examples for the other dimensions are also attached:

a- "ordinal" dimension: ordinal bug

b- "number" dimension: This example shows bugs in more the one dimension, number dim is one of them. number bug

Contributor guide

Tech stack
None
Domain
backend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
basic regexArabic language
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate the Arabic regex patterns in Duckling's time, ordinal, number, and duration dimensions. Look at the relevant source files in the repository (likely under `duckling/`). The issue shows partial word matching; consider adding word boundaries. Check the maintainer comments for any suggested approach.