Error during compilation: missing "lexer.h"

The failure is in yara/libyara/grammar.c, which includes yara/parser.h and requires lexer.h; inspect how these headers are packaged and exposed during pip compilation. Reproduce the macOS installation command, then verify the build completes without the missing-header error.

Hello,
I am trying to install yara-python on my mac (from pypi, 4.0.2 version) which complaints about missing "lexer.h" header file during compilation. I found out that this header file is provided in the main yara repository in the include files but is not being copied to OS as part of installation.

I attempted to install the published homebrew package (latest 4.0.2) as well as force rebuild from sources. Here is the output and content of the installed yara:

==> Downloading https://github.com/VirusTotal/yara/archive/v4.0.2.tar.gz
==> Downloading from https://codeload.github.com/VirusTotal/yara/tar.gz/v4.0.2
      # -=#=- #     #
==> ./bootstrap.sh
==> ./configure --prefix=/usr/local/Cellar/yara/4.0.2 --enable-dotnet --enable-cuckoo --enable-magic --enable-macho --enable-dex --with-crypto
==> make install
🍺  /usr/local/Cellar/yara/4.0.2: 46 files, 1.6MB, built in 38 seconds
λ PC ~ → brew list yara -v
find /usr/local/Cellar/yara/4.0.2 -not -type d -print
/usr/local/Cellar/yara/4.0.2/INSTALL_RECEIPT.json
/usr/local/Cellar/yara/4.0.2/bin/yara
/usr/local/Cellar/yara/4.0.2/bin/yarac
/usr/local/Cellar/yara/4.0.2/.brew/yara.rb
/usr/local/Cellar/yara/4.0.2/AUTHORS
/usr/local/Cellar/yara/4.0.2/include/yara.h
/usr/local/Cellar/yara/4.0.2/include/yara/error.h
/usr/local/Cellar/yara/4.0.2/include/yara/utils.h
/usr/local/Cellar/yara/4.0.2/include/yara/exefiles.h
/usr/local/Cellar/yara/4.0.2/include/yara/proc.h
/usr/local/Cellar/yara/4.0.2/include/yara/strutils.h
/usr/local/Cellar/yara/4.0.2/include/yara/compiler.h
/usr/local/Cellar/yara/4.0.2/include/yara/base64.h
/usr/local/Cellar/yara/4.0.2/include/yara/types.h
/usr/local/Cellar/yara/4.0.2/include/yara/limits.h
/usr/local/Cellar/yara/4.0.2/include/yara/ahocorasick.h
/usr/local/Cellar/yara/4.0.2/include/yara/parser.h
/usr/local/Cellar/yara/4.0.2/include/yara/re.h
/usr/local/Cellar/yara/4.0.2/include/yara/threading.h
/usr/local/Cellar/yara/4.0.2/include/yara/bitmask.h
/usr/local/Cellar/yara/4.0.2/include/yara/libyara.h
/usr/local/Cellar/yara/4.0.2/include/yara/stream.h
/usr/local/Cellar/yara/4.0.2/include/yara/sizedstr.h
/usr/local/Cellar/yara/4.0.2/include/yara/modules.h
/usr/local/Cellar/yara/4.0.2/include/yara/scanner.h
/usr/local/Cellar/yara/4.0.2/include/yara/filemap.h
/usr/local/Cellar/yara/4.0.2/include/yara/stopwatch.h
/usr/local/Cellar/yara/4.0.2/include/yara/atoms.h
/usr/local/Cellar/yara/4.0.2/include/yara/stack.h
/usr/local/Cellar/yara/4.0.2/include/yara/exec.h
/usr/local/Cellar/yara/4.0.2/include/yara/object.h
/usr/local/Cellar/yara/4.0.2/include/yara/integers.h
/usr/local/Cellar/yara/4.0.2/include/yara/mem.h
/usr/local/Cellar/yara/4.0.2/include/yara/rules.h
/usr/local/Cellar/yara/4.0.2/include/yara/arena.h
/usr/local/Cellar/yara/4.0.2/include/yara/notebook.h
/usr/local/Cellar/yara/4.0.2/include/yara/hash.h
/usr/local/Cellar/yara/4.0.2/include/yara/scan.h
/usr/local/Cellar/yara/4.0.2/README.md
/usr/local/Cellar/yara/4.0.2/COPYING
/usr/local/Cellar/yara/4.0.2/lib/pkgconfig/yara.pc
/usr/local/Cellar/yara/4.0.2/lib/libyara.a
/usr/local/Cellar/yara/4.0.2/lib/libyara.4.dylib
/usr/local/Cellar/yara/4.0.2/lib/libyara.dylib
/usr/local/Cellar/yara/4.0.2/share/man/man1/yara.1
/usr/local/Cellar/yara/4.0.2/share/man/man1/yarac.1

As it can be seen header files from include/yara are being copied however the one file "lexer.h" that python-yara errors on is missing although it is present in the repository when I checked manually. Is there some compile/install flags I need to add to have that file included/copied on my OS?

Copy paste of error when doing pip install yara-python:

      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_GNU_SOURCE=1 -DUSE_MACH_PROC=1 -Iyara/libyara/include -Iyara/libyara/ -I. -I/usr/local/opt/openssl/include -I/opt/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/tcl-tk/include -I/Users/intense/Documents/Workspace/aura/venv/include -I/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yara/libyara/filemap.c -o build/temp.macosx-11-x86_64-3.9/yara/libyara/filemap.o -std=c99
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -D_GNU_SOURCE=1 -DUSE_MACH_PROC=1 -Iyara/libyara/include -Iyara/libyara/ -I. -I/usr/local/opt/openssl/include -I/opt/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/tcl-tk/include -I/Users/intense/Documents/Workspace/aura/venv/include -I/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yara/libyara/grammar.c -o build/temp.macosx-11-x86_64-3.9/yara/libyara/grammar.o -std=c99
      In file included from grammar.y:51:
      /usr/local/include/yara/parser.h:34:10: fatal error: 'lexer.h' file not found
      #include "lexer.h"
               ^~~~~~~~~
      1 error generated.
      error: command '/usr/local/opt/llvm/bin/clang' failed with exit code 1

I see also that there is -Iyara/libyara/ -Iyara/libyara/include but shouldn't then the include header be "yara/lexer.h" ?