Reporting an issue with util.SizeEstimator, as requested.
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Active
- Domain
- backend, distributed-systems
Research direction
Start by identifying whether the reported IndexError comes from PySpark's Jupyter kernel event loop or from Spark's SizeEstimator warning; the report includes both outputs but no reproduction steps or expected behavior. Read the traceback entry points in ipykernel_launcher.py and Tornado's asyncio loop, then check whether the warning is related. Done means reproducing the failure and establishing which component owns it before proposing a fix.
Written by the indexing model from the issue text.
Description
WARNING: An illegal reflective access operation has occurred (0 + 8) / 8]
WARNING: Illegal reflective access by org.apache.spark.util.SizeEstimator$ (file:/usr/lib/spark/jars/spark-core_2.12-3.5.3.jar) to field java.lang.ref.Reference.referent
WARNING: Please consider reporting this to the maintainers of org.apache.spark.util.SizeEstimator$
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Traceback (most recent call last):==================================(4 + 1) / 4]
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/opt/conda/miniconda3/lib/python3.11/site-packages/ipykernel_launcher.py", line 18, in
app.launch_new_instance()
File "/opt/conda/miniconda3/lib/python3.11/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/miniconda3/lib/python3.11/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/miniconda3/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/miniconda3/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
self._run_once()
File "/opt/conda/miniconda3/lib/python3.11/asyncio/base_events.py", line 1921, in _run_once
handle = self._ready.popleft()
^^^^^^^^^^^^^^^^^^^^^
IndexError: pop from an empty deque
- Dominant language
- Scala
- Stars
- 44.2k
- Forks
- 29.5k
- Avg merge
- 8h 43m
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/spark
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Maintainers usually reply within 2 days
-
[SQL][CONNECT] SparkConnectClient may drop metadata (auth headers) on retried/reattached RPCs if metadata is a non-list iterablePossibly taken @zainmirzaaa claimed this 60 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 2 days
-
[CORE] bug report: collision of resource profile idPossibly taken @ybapat claimed this 86 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 82/100
apache/spark#57498 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/spark#57209 · 7 comments ·
Maintainers usually reply within 2 days
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Maintainers usually reply within 2 days
Similar issues
-
Difficulty 1/5 1-3 hours Newbie friendliness 91/100
com-vitthalmirji/toon4s#117 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
zio/zio-blocks#1780 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
digital-asset/participant-query-store#168 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
propensive/soundness#2294 ·
Maintainers usually reply within 1 day