POST: purl2sym, extracting symbols from sources for fun and for profit
还没有人认领这个 Issue。
评估
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 35/100
- Issue 类型
- 文档
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 领域
- content, documentation
调研方向
该仓库被描述为基于 Docusaurus/Markdown，但没有指定目标文件或内容目录。首先定位网站现有的文章结构，并将其与链接的 purl2sym 和 Symbols and Strings in PURLdb 参考资料进行比较。完成的标准是：将此草稿发布到网站中适当的位置，并保留其中的链接和技术性论断。
由索引模型根据 Issue 内容生成。
描述
Lightweight reverse-engineering of open source binaries with purl2sym
TL;DR
- Got binary code? Got source code?
- Need to know what open source packages it came from?
- Use purl2sym to extract and then index symbols, like function names, and strings.
- Extract strings from the binary and match them back to actual source packages.
Context
As an open source developer, I further rely on open source every day. But what happens when that code is shipped as a binary, like when embedded in devices (a car, a phone, or a toaster) or in container images where the original corresponding source code is not present?
You get a blob. Maybe a stripped ELF file. How can you figure out what is inside, what is it made off, what is the source, and where is the "open" source?
This is where purl2sym comes in. It is one of the AboutCode https://aboutcode.org projects, funded by NLnet https://nlnet.nl/project/purl2sym and it extracts and indexes symbols from source code so you can match these back to binary symbols of open source packages, keyed by PURL https://github.com/package-url/purl-spec. It can help turn otherwise mostly obscure and opaque symbols and strings found in binaries into traceable references to specific open source code packages and specific versions.
Problems
If you have ever faced some of these use cases, then purl2sym could help:
- Analyzed a binary firmware blob to find which open source packages it embeds
- Tried to comply with FOSS license obligations post-compilation, to find what the actual code that is effectively compiled and shipped (as opposed to the 10x more code used in development and testing)
- Hunted down the source code of a vulnerability found in a binary
- Built SBOMs, working from compiled codebase
Solution
purl2sym extract symbols from source code and builds indexes of symbols and strings extracted from multiple tech stacks and programing languages, like C, C++, Java, Python JavaScript, packaged in Linux distros like Debian or Alpine, or application packages like PyPI, Maven and more. You can fetch the symbols by PURL (Package-URL) and cross-reference these with binary symbols to get back matches to potential packages and versions indexed in the PrulDB.
How it works
purl2sym is a combo of low level libraries, scripted pipelines and high level APIs. When you call the high level REST API, you request the symbols for a PURL, and the tool:
- Downloads and extracts the source package
- Extracts symbols and strings from the code using tools like
gettext,treesitter, orpygments. - Stores those symbols keyed by PURL (e.g.,
pkg:deb/debian/[email protected]). - Returns a JSON doc with the symbols and metadata for further integration, alongside hashes and fingerprints.
With this, I can analyze unknown binary blobs by extracting their symbols and finding their original sources, using symbol matching. Or determine if the symbols of a code vulnerability may be present or used in a given source or binary. (Assuming we know which code segment is the trigger for a vulnerability, but this another thing, handled in another project)/
It is designed to work with and power binary analysis tools like BANG https://github.com/armijnhemel/binaryanalysis-ng/ that can then detect reused code chunks across binaries.
In practice
Let’s say you are working with the firmware blob of a commercial device. It includes a statically linked C executable, but it is not obvious what libraries are linked in the binary. You could pull a few function names using strings (or commoncode's string functions) , or maybe extract a specific ELF section.
You can feed that into a tool powered by purl2sym. Voila: it will suggests that OpenSSL 1.1.1d from Debian is linked in the exe, and you can further lookup in the PurlDB for complete licensing metadata and in Vulnerable for CVEs. You now have a clear path towards automating a security and license compliance.
Get Started
- Project homepage: https://nlnet.nl/project/purl2sym
- Docs on symbol extraction: Symbols and Strings in PURLdb
- Install ScanCode.io to run locally
- Binary analysis with BANG: https://github.com/nexB/bang
- 主要语言
- JavaScript
- 星标
- 9
- 派生
- 17
- 平均合并
- 7 小时 55 分钟
- 30 天内合并 PR
- 2
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
aboutcode-org/www.aboutcode.org 的其他 Issue
-
Making all post button.可能已有人在做 @sarthaklole 于 71 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
aboutcode-org/www.aboutcode.org#117 · 3 条评论 ·
-
WWW: Reduce frequency of update-releases可能重新可做 @johnmhoran 于 52 天前认领，目前没有进行中的 PR。 未关闭ci-workflow
难度 2/5 1-3 小时 新手友好度 45/100
aboutcode-org/www.aboutcode.org#180 · 已指派 1 人 ·
-
DOCS: Create new AboutCode stack diagram(s)可能重新可做 @hixvikas 于 53 天前认领，目前没有进行中的 PR。 未关闭documentation
难度 5/5 一周以上 新手友好度 35/100
aboutcode-org/www.aboutcode.org#166 · 1 条评论 ·
-
WWW: Create About top level page?可能已有人在做 @pombredanne 于 155 天前认领。 未关闭website
难度 2/5 1-3 小时 新手友好度 35/100
aboutcode-org/www.aboutcode.org#164 · 1 个 reaction · 已指派 1 人 ·
-
WWW: Add Initiatives and grant Projects可能重新可做 @mjherzog 于 241 天前认领，目前没有进行中的 PR。 未关闭enhancement needs design website
难度 5/5 一周以上 新手友好度 25/100
aboutcode-org/www.aboutcode.org#141 · 已指派 1 人 ·
查看 aboutcode-org/www.aboutcode.org 的全部 Issue
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 88/100
-
tech-debt
难度 2/5 1-3 小时 新手友好度 72/100
srikanth235/centraid#1155 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 73/100
microsoft/monaco-editor#5563 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复