NetFlow v9: zero OUT_BYTES/OUT_PKTS (FreeBSD ng_netflow/OPNsense) overwrite real IN_BYTES/IN_PKTS — bytes always 0

[thedancingdeveloper]

Describe the bug

With a FreeBSD ng_netflow exporter (OPNsense's built-in Netflow), every decoded flow has bytes: 0 and packets: 0 even though the exporter sends correct counters on the wire.

This is the same root cause as #186 (closed without a code change): ng_netflow's v9 template places OUT_BYTES (23) / OUT_PKTS (24) after IN_BYTES (1) / IN_PKTS (2), and always exports the OUT counters as 0 (its flows are unidirectional). producer_nf.go assigns both field pairs to the same output members:

case netflow.NFV9_FIELD_IN_BYTES:
    DecodeUNumber(v, &(flowMessage.Bytes))
...
case netflow.NFV9_FIELD_OUT_BYTES:
    DecodeUNumber(v, &(flowMessage.Bytes))

Fields are processed in record order, so the trailing zero OUT_BYTES overwrites the real IN_BYTES on every record.

Wire evidence

Captured the template broadcast from OPNsense 25.x and decoded it manually. IPv4 template (ID 256), fields in order (type, len):

(8,4) (12,4) (15,4) (10,2) (14,2) (2,4) (1,4) (24,4) (23,4) (22,4) (21,4)
(7,2) (11,2) (6,1) (4,1) (5,1) (16,4) (17,4) (9,1) (13,1)

Note IN_PKTS(2), IN_BYTES(1) followed immediately by OUT_PKTS(24), OUT_BYTES(23). Decoding 3,891 data records from the same capture: 100% had nonzero IN_BYTES and 100% had zero OUT_BYTES. goflow2's JSON output for the same traffic: "bytes":0,"packets":0 on every flow.

Why the documented mapping workaround cannot fix bytes

In #186 a custom mapping was suggested as the fix, and a later commenter reported they could not make it work no matter the mapping — that is expected: ConvertNetFlowDataSet calls MapCustomNetFlow before the builtin switch for each field, so a mapping targeting bytes is always followed by the builtin zero-assignment from field 23 later in the record. Mapping can only rescue the data into new fields, e.g.:

formatter:
  fields: [ ..., in_bytes, in_packets ]
  protobuf:
    - { name: in_bytes,   index: 3001, type: varint }
    - { name: in_packets, index: 3002, type: varint }
netflowv9:
  mapping:
    - { field: 1, destination: in_bytes }
    - { field: 2, destination: in_packets }

This works (verified), but leaves the standard bytes/packets fields wrong for one of the most common home/lab firewall exporters.

Proposed fix

In the OUT_BYTES / OUT_PKTS cases, skip the assignment when the decoded value is 0 and the destination already holds a nonzero value (i.e. prefer nonzero IN_*). That preserves current behaviour for egress-only exporters while fixing exporters that send both directions with one side zeroed. Happy to open a PR if this approach is acceptable.

To reproduce

OPNsense (or any FreeBSD ng_netflow) → NetFlow v9 export → goflow2 -format=json: all flows log bytes:0, packets:0.

Expected behaviour

bytes/packets reflect IN_BYTES/IN_PKTS when the OUT counters are zero.

Version

netsampler/goflow2:latest (current as of 2026-08); the assignment is unchanged in main (producer/proto/producer_nf.go).

[thedancingdeveloper]

For reference: an implementation of exactly this fix already exists in a fork — VigilProtector/goflow2@3781976 ("prevent OUT_BYTES/OUT_PKTS from overwriting IN values", guards the OUT_* assignment with if flowMessage.Bytes == 0). It references #186 but was never submitted as a PR here. Happy to submit it (or an equivalent) if the approach is acceptable.