Problem resolving some symbols when using the library in an Android C++ project (I am compiling using ndk)
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 技术栈
- android, cpp
调研方向
先从 Android 示例和库链接设置开始,然后检查 src/sentencepiece_tokenizer.cc、tokenizers_cpp.h 以及生成的静态 .a 文件。重现 NDK 链接命令,并比较哪些 SentencePiece 符号可用且已链接。Android C++ 示例能够成功链接并运行,且不再出现报告的未定义符号错误,即表示完成。
由索引模型根据 Issue 内容生成。
描述
I was able to successfully compile the library but when I use it as indicated in the example folder I am having the following errors:
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::SentencePieceProcessor()
referenced by sentencepiece_tokenizer.cc:18 (src/sentencepiece_tokenizer.cc:18)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::LoadFromSerializedProto(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >)
referenced by sentencepiece_tokenizer.cc:19 (src/sentencepiece_tokenizer.cc:19)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::util::Status::~Status()
referenced by sentencepiece_tokenizer.cc:19 (src/sentencepiece_tokenizer.cc:19)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced 2 more times
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::~SentencePieceProcessor()
referenced by sentencepiece_tokenizer.cc:20 (src/sentencepiece_tokenizer.cc:20)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:16 (src/sentencepiece_tokenizer.cc:16)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::~SentencePieceTokenizer()) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::Encode(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >, std::__ndk1::vector<int, std::__ndk1::allocator >*) const
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::util::Status::IgnoreError()
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:30 (src/sentencepiece_tokenizer.cc:30)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator >*) const
referenced by sentencepiece_tokenizer.cc:30 (src/sentencepiece_tokenizer.cc:30)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::GetPieceSize() const
referenced by sentencepiece_tokenizer.cc:35 (src/sentencepiece_tokenizer.cc:35)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::GetVocabSize()) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::IdToPiece(int) const
referenced by sentencepiece_tokenizer.cc:40 (src/sentencepiece_tokenizer.cc:40)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::IdToToken(int)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::PieceToId(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >) const
referenced by sentencepiece_tokenizer.cc:42 (src/sentencepiece_tokenizer.cc:42)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::TokenToId(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
When I check inside the library the symbols are properly defined.
In my code I am just doing the same as in the example folder, so I am not invoking directly the symbols that are not recognized. The ones that I am using (FromBlobSentencePiece, for example) are correctly identified. What could be the error?
One things which is curious for me is: why the compiler of my program is complaining about the src/sentencepiece_tokenizer.cc file if I am just using the static library (the .a file) through the tokenizers_cpp.h file provided by the library?
- 主要语言
- C++
- 星标
- 514
- 派生
- 132
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
mlc-ai/tokenizers-cpp 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 68/100
mlc-ai/tokenizers-cpp#96 · 1 条评论 · 1 个 reaction ·
-
难度 2/5 1-3 小时 新手友好度 62/100
mlc-ai/tokenizers-cpp#88 · 2 个 reaction ·
-
难度 3/5 1-2 天 新手友好度 55/100
mlc-ai/tokenizers-cpp#95 ·
-
难度 3/5 1-2 天 新手友好度 35/100
mlc-ai/tokenizers-cpp#94 · 1 个 reaction ·
-
难度 4/5 3-5 天 新手友好度 35/100
mlc-ai/tokenizers-cpp#91 ·
查看 mlc-ai/tokenizers-cpp 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 72/100
KhronosGroup/Vulkan-Tutorial#534 ·
-
难度 2/5 1-3 小时 新手友好度 65/100
维护者通常 2 天内回复
-
bug IO
难度 2/5 1-3 小时 新手友好度 65/100
MrNeRF/LichtFeld-Studio#3824 ·
维护者通常 1 天内回复
-
pnnx: manual tanh GELU fusion loses the approximation mode可能已有人在做 @ktz03 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
Tencent/ncnn#7175 · 1 个 reaction ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 77/100
throneproj/Throne#2043 ·
维护者通常 2 天内回复