Inner-node communication doesn't survive reboots on OCI
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- linux, nginx, python
- Domain
- cloud, infrastructure, networking
Research direction
Start with the OCI startup logic in oci-core/nomad.py and the reboot handling in oci-core/files/nomad-rc.local, focusing on when enp1s0 is brought down. Observe the NIC and IPv6 routing state across an OCI maintenance reboot and compare it with the documented traffic path through nginx-dc1. Done means the secondary NIC stays down and inner-cluster IPv6 traffic remains reliable after reboots.
Written by the indexing model from the issue text.
Description
When OCI does maintenance on an node, it reboots the node. This works fine for everything except IPv6.
For IPv6, we cheat a bit on OCI, and we end up with two VNICs on the same IPv6 segment. This is needed to work around some IPv6 issues with OCI, and its inability to route IPv6 traffic for multiple smaller segments to/from nodes.
On initial startup [1] and reboot [2], we run:
ip link set enp1s0 down
To ensure both nics don't conflict with each other. They both announce the same IPv6, so it is important only one is actually running. The other exists purely so OCI routes certain IPv6 traffic to this node.
But sometimes this breaks, and the second nic comes online again. Depending on internal priorities of the Linux kernel, this can mean the IPv6 routing actually breaks down, and only parts of the cluster can route traffic to/from nodes. This results in partial outages.
The traffic-flow is something like this:
Cloudflare -> Cloudflare ZeroTrust -> Cloudlfare Tunnel -> nginx-dc1 -> (actual pod).
The last two use this inner-cluster communication over IPv6. And if this traffic crosses nodes, and IPv6 is not in a functional state, traffic gets lost.
Not sure what a proper solution is; manually shutting the NIC down most often resolve the problem for months. But after a maintenance of OCI, it can come back online. It might be that the reboot script just does it too early, or something silly like that.
- Dominant language
- Python
- Stars
- 12
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Similar issues
-
component/fhir type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
OpenConceptLab/ocl_issues#3026 ·
-
ODS datetime imports reject fractional seconds and exports drop themPossibly taken @IlyaKhalafi claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
TauricResearch/TradingAgents#1693 ·
Maintainers usually reply within 2 days
-
requests support
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
UKGovernmentBEIS/inspect_ai#6537 ·
Maintainers usually reply within 1 day