Hacktoberfest 2026:维护者为十月标记出来的 issue,仍然开放、适合新手。 浏览 Hacktoberfest issue

How to build in android platform?

未关闭
#56 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
25/100
Issue 类型
文档
描述清晰度
需要澄清
活跃度
停滞
技术栈
android, cmake, cpp

调研方向

从报告中的 CMakeLists.txt 及其 target_link_libraries 条目开始,然后检查 libtokenizers_cpp.a 和相关库是如何为 Android 构建和链接的。使用列出的未定义 tokenizers_* 符号来追踪缺失的依赖项;当 Android 构建说明和所需的链接输入都清晰明确,且示例目标成功链接时,即表示完成。

由索引模型根据 Issue 内容生成。

描述

int main() {
std::string tokenizer_json_path = "../tokenizer.json", path;
std::unique_ptrtokenizers::Tokenizer m_tokenizer;

path = LoadBytesFromFile(tokenizer_json_path);
cout<<"load file success !"<<endl;

m_tokenizer = tokenizers::Tokenizer::FromBlobJSON(path);

Here is my CMakeLists.txt info:
{

cmake_minimum_required(VERSION 3.18)

project(target)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED True)

aux_source_directory(. SRC_FILES)

link_directories(libs)

include_directories(include)

add_executable(target ${SRC_FILES})
target_link_libraries(target
${PROJECT_SOURCE_DIR}/libs/libsentencepiece.a
${PROJECT_SOURCE_DIR}/libs/libtokenizers_c.a
${PROJECT_SOURCE_DIR}/libs/libtokenizers_cpp.a
)

}

When I test, met the following issues:

/usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::Tokenizer::FromBlobJSON(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)': huggingface_tokenizer.cc:(.text+0x46): undefined reference to tokenizers_new_from_str'
/usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::Tokenizer::FromBlobByteLevelBPE(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)': huggingface_tokenizer.cc:(.text+0x13e): undefined reference to byte_level_bpe_tokenizers_new_from_str'
/usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::HFTokenizer::~HFTokenizer()': huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizerD2Ev[_ZN10tokenizers11HFTokenizerD5Ev]+0x37): undefined reference to tokenizers_free'
/usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::HFTokenizer::Encode(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)': huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer6EncodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb[_ZN10tokenizers11HFTokenizer6EncodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb]+0x69): undefined reference to tokenizers_encode'
/usr/bin/ld: huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer6EncodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb[_ZN10tokenizers11HFTokenizer6EncodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb]+0xbc): undefined reference to tokenizers_free_encode_results' /usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::HFTokenizer::EncodeBatch(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&, bool)':
huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer11EncodeBatchERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EEb[_ZN10tokenizers11HFTokenizer11EncodeBatchERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EEb]+0x203): undefined reference to tokenizers_encode_batch' /usr/bin/ld: huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer11EncodeBatchERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EEb[_ZN10tokenizers11HFTokenizer11EncodeBatchERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EEb]+0x344): undefined reference to tokenizers_free_encode_results'
/usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::HFTokenizer::Decode[abi:cxx11](std::vector<int, std::allocator<int> > const&, bool)': huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer6DecodeB5cxx11ERKSt6vectorIiSaIiEEb[_ZN10tokenizers11HFTokenizer6DecodeB5cxx11ERKSt6vectorIiSaIiEEb]+0x62): undefined reference to tokenizers_decode'
/usr/bin/ld: huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer6DecodeB5cxx11ERKSt6vectorIiSaIiEEb[_ZN10tokenizers11HFTokenizer6DecodeB5cxx11ERKSt6vectorIiSaIiEEb]+0x7d): undefined reference to tokenizers_get_decode_str' /usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::HFTokenizer::GetVocabSize()':
huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer12GetVocabSizeEv[_ZN10tokenizers11HFTokenizer12GetVocabSizeEv]+0x32): undefined reference to tokenizers_get_vocab_size' /usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::HFTokenizer::IdToTokenabi:cxx11':
huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer9IdToTokenB5cxx11Ei[_ZN10tokenizers11HFTokenizer9IdToTokenB5cxx11Ei]+0x3e): undefined reference to tokenizers_id_to_token' /usr/bin/ld: ../libs/libtokenizers_cpp.a(huggingface_tokenizer.cc.o): in function tokenizers::HFTokenizer::TokenToId(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)':
huggingface_tokenizer.cc:(.text._ZN10tokenizers11HFTokenizer9TokenToIdERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN10tokenizers11HFTokenizer9TokenToIdERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x58): undefined reference to `tokenizers_token_to_id'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/target.dir/build.make:116: target] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/target.dir/all] Error 2

主要语言
C++
星标
514
派生
132
PR 合并指标
30 天内没有已合并 PR

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

mlc-ai/tokenizers-cpp 的其他 Issue

查看 mlc-ai/tokenizers-cpp 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。