[Python][JS] Examples of serializing arrow from python through ipywidgets to JS
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python, typescript
- Domain
- documentation
Research direction
Start with the Python serialization snippet and the TypeScript base64ToBytes/tableFromIPC path, then compare them with the working JavaScript base64 case. Done is a documented, reproducible Python-to-JS Arrow table example covering the ipywidgets or base64 path and explaining the expected result.
Written by the indexing model from the issue text.
Description
Describe the usage question you have. Please include as many useful details as possible.
I do not know how to use ipywidgets traitlets.Bytes or even base64 encoding to generate an arrow table that will reify in JS.
lets start with Base64
import pyarrow as pa
import base64
df = pd.DataFrame({'a':[10,50], 'b': ['paddy', 'margaret']})
table = pa.Table.from_pandas(df)
batch = table.to_batches()[0]
pabuffer = batch.serialize()
pabuffer
b64_table = base64.b64encode(pabuffer.to_pybytes())
print(base_64_table)
given the following JS code, that string doesn't reify properly
export function base64ToBytes(base64:string) {
const binString = atob(base64);
//@ts-ignore
return Uint8Array.from(binString, (m) => m.codePointAt(0));
}
const b64bytes = base64ToBytes(base64table)
const t2 = tableFromIPC(b64bytes);
I get an error of TypeError: this.schema is undefined
I can get an arrow table to reify when I do base64 in JS and use that string.
I have made a sample repo to play with, benchmark, and document dataframe serialization. https://github.com/paddymul/df_cereal
I'm eager to collaborate and help with documentation for using arrow-js. I think it solves a lot of serialization problems for performance and typing vs json. It's a little hard to approach as is, because the docs are written for someone who understands arrow, vs someone who knows less about arrow, but wants to use it for serialization.
Component(s)
JavaScript, Python
- Dominant language
- TypeScript
- Stars
- 112
- Forks
- 23
- Avg merge
- 17h 55m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-js
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 50/100
Similar issues
-
bug(cli): hapi doctor inline-media prints a fabricated B:\ helper-script path in packaged installs Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Crush Open
Difficulty 1/5 Under an hour Newbie friendliness 85/100
catppuccin/catppuccin#3125 ·
-
Add a SECURITY.md Open
Difficulty 1/5 Under an hour Newbie friendliness 90/100
ElementsProject/cln-application#167 · 1 comment · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Quantco/pnpm-licenses#17 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100