IPFIX flow selection (IE 390 and the 391-399 family) is not modelled at all

Open
  • Is something wrong?
#596 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
java
Domain
backend, data

Research direction

Start with IpfixRawFlow, SelectorReport, and the shipped ipfix-information-elements.xml, then review #581, #584, and #594 to understand the existing packet-selection model and provenance decisions. Resolve whether flow-selection data belongs in samplingInterval or a separate field, how ratios should be derived, and how packet and flow selection compose; done requires an agreed design and evidence from a conformant exporter before implementation.

Written by the indexing model from the issue text.

Description

enhancement priority: low

riptide models packet selection and not flow selection. #594 established the packet-selection path properly — IE 302 selectorId on the record, RFC 5476 §6.5.2 Selector Reports in options records, IE 304 selectorAlgorithm and its parameters. The flow-selection family one layer up is bound nowhere.

None of IE 390-399 appears on IpfixRawFlow:

390  flowSelectorAlgorithm            394  selectorIDTotalFlowsObserved
391  flowSelectedOctetDeltaCount      395  selectorIDTotalFlowsSelected
392  flowSelectedPacketDeltaCount     396  samplingFlowInterval
393  flowSelectedFlowDeltaCount       397  samplingFlowSpacing
                                      398  flowSamplingTimeInterval
                                      399  flowSamplingTimeSpacing

An exporter running an Intermediate Flow Selection Process therefore reports its flows at whatever rate its packet selection states, or at 1, with no signal that most of its flows were discarded before export.

This is not a hypothetical gap, it is a regression the code used to paper over

Before #594, fromSelectorAlgorithm() read samplingFlowInterval (396), samplingFlowSpacing (397), flowSamplingTimeInterval (398) and flowSamplingTimeSpacing (399) — the flow-selection family — while switching on IE 304, which is packet selection. The two never appear together in a conformant export, so those lookups always missed and the branch fabricated a rate from its defaults. #594 removed the conflation.

The result is that riptide now reads the flow-selection elements correctly, which is to say not at all. That is an improvement over reading them under the wrong algorithm, and it leaves a real capability unimplemented.

The registry exists locally, unlike the one #584 was missing

The shipped ipfix-information-elements.xml does carry ipfix-flowselectoralgorithm, with the same ten values as the packet registry:

0 Unassigned                       5 Property Match Filtering
1 Systematic count-based Sampling  6 Hash-based Filtering using BOB
2 Systematic time-based Sampling   7 Hash-based Filtering using IPSX
3 Random n-out-of-N Sampling       8 Hash-based Filtering using CRC
4 Uniform probabilistic Sampling   9 Flow-state Dependent

This is the opposite of the situation in #584, where IE 304's registry was absent from the file and both hand-written switches had drifted. Here SelectorReport's existing partition — which algorithms express a ratio and which do not — transfers directly, because the value sets are identical.

What makes this genuinely different from packet selection

Not just "the same thing one layer up". Three things change:

The counters are exact, and the records are missing. Under packet sampling a surviving flow record undercounts its own bytes. Under flow selection a surviving record's counters are correct and whole flows are simply absent. Multiplying by the ratio gives an unbiased estimate of aggregate volume either way, but a per-flow question — top talkers, whether host X spoke to host Y — cannot be answered at all for the discarded flows, and no multiplier repairs that.

The two compose. An exporter can sample packets and select flows. The true correction is the product of both ratios, and riptide has one samplingInterval column. Modelling only one silently drops the other.

The exporter can tell us directly. IEs 394 and 395 (selectorIDTotalFlowsObserved / Selected) are an observed ratio rather than a configured one, which is a stronger signal than any parameter-derived figure. 391-393 give the selected counters explicitly. Neither has an analogue in the packet-selection path.

Open questions

  1. Should samplingInterval carry the product of packet and flow selection, or should flow selection get its own column? One column is simpler and loses the ability to say which process discarded what.
  2. Should a flow-selection ratio be derived from parameters (396-399), or measured from 394/395 where the exporter states both? The measured ratio is better evidence, and the provenance vocabulary has no value that distinguishes it.
  3. Does derived cover a rate computed from a flow selector, or does that need its own provenance? See #581, which is already asking a related question about carrying provenance into the rollups.
  4. Is there a conformant exporter to test against? #584's survey found no collector reading these and no open-source exporter writing them. Same evidence problem, and the same reason not to build it speculatively.

Recommendation

File and leave. No exporter on the reference fleet uses flow selection, and #584's survey found nobody implementing the flow-selection family at all — not nfdump, Akvorado, goflow2, vflow or pmacct. Building it now means a second round of writing an unverifiable implementation from an RFC, which is exactly how #584's four defects got there.

Worth doing when an exporter turns up that sends IE 390. Worth reading now only for question 1, which affects how samplingInterval should be understood and therefore touches #581.

Split off from #584, which shipped the packet-selection half in #594. Related: #467, #581, #585.

Dominant language
Java
Stars
2
Forks
0
Avg merge
1d 1h
Merged PRs (30d)
52

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Riptide-Labs/riptide

All issues in Riptide-Labs/riptide

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.