Clarify no-op usage

[nebkat]

The no-op value type is somewhat ambiguous.

---

My current interpretation is:

A. In an array - the element is skipped

[[]
    [#][i][2]
    [i][1]
    [N]
    [N]
    [i][2]

[1, 2]

Note that the container count doesn't include the no-op elements.

B. In an object - the key is discarded

[{]
    [#][i][2]
    [i][3][foo][i][1]
    [N]
    [i][3][bar][i][2]

{"foo": 1, "bar": 2}

Should the following also be allowed? The merits of it are somewhat questionable, as there is no point in including the key if the value is going to cause it to be removed.

[{]
    [#][i][2]
    [i][3][foo][i][1]
    [i][3][bar][N]

{"foo": 1}

C. Top level N - Error or symbol/sentinel value

In languages where it is available: Symbol('noop') (JavaScript, Dart).

Otherwise throw a dedicated error: 'N' can only occur in arrays/objects (distinct from an unexpected byte error!), and ignore if processing a stream.

Importantly, do not skip N before/after a value is found: bjdata.decode("NTN") != bjdata.decode("T").

D. In an un-delimited sequence of BJData objects / an implicit stream of values - the entry is ignored:

Sequence of BJData values in a file or streamed over network - implicitly wrapped in [ and ] so effectively the same as A.

[T]
[i][4]
[N]
[F]

true
4
false

---

Another way to interpret it is at the tokenizer level, where it could appear anywhere a token is expected i.e. only excluding strongly-typed contexts where it is impossible to differentiate between the value 0x4E and N:

Surrounding a value:

[N][S][i][3][foo][N][EOF]

"foo" // Not a stream, but doesn't complain because [N] are ignored

When waiting for a length:

[S][N][i][3][foo]

"foo"

When waiting for a count:

[[]
    [$][U]
    [N]
    [#][N][i][1]
    [5]

[5]

Within an array:

[[]
    [#][i][2]
    [i][1]
    [N]
    [N]
    [i][2]

[1, 2]

nlohmann/json exhibits some of these behaviors.

---

Don't think the latter interpretation is ever what was intended, but we would certainly benefit from being more explicit.

To summarize:

1. Can N appear in arrays? ✅ This is the main purpose
2. Can N appear in objects before keys? ✅ Also potentially useful
3. Can N appear in objects after keys? ❌ Seems redundant
4. Can N appear at top-level? ❌ It is not meaningful in that context. Throw error/special value, and handle in streams.
5. Can N appear in special contexts (between [/$/#, S/i, etc)? ❌ Seems redundant
6. Does N contribute towards container count? ❌ We might know we have to produce 10 values, but have to send keep-alives in between them because processing is slow. Could also ban N in counted containers.
7. Are leading/trailing N ignored when looking for a value? ❌

If we can agree on the expected behavior I can make a PR with the appropriate spec changes.

[fangq]

@nebkat, the no-op/N marker was inherited from UBJSON, see spec

https://ubjson.org/type-reference/value-types/#noop

my initial understanding was that, theoretically, N is a filler it can appear anywhere, but it will be skipped. But practically, implementing parsers that can detect and skip N anywhere it appears is quite cumbersome and adds unnecessary complexity with diminished returns.

The primary use of N is in the streaming between elements of array elements and object keys. Therefore, UBJSON spec highlighted the use of fillers in an array, but did not exclude other places.

From the Python UBJSON implementation (py-ubjson), it appears that NOOP are only handled inside arrays and objects

https://github.com/search?q=repo%3ANeuroJSON%2Fpybj%20TYPE_NOOP&type=code

however, in my MATLAB/Octave implementation, I skip it in more places

https://github.com/NeuroJSON/jsonlab/blob/da0bdeed1b3d6d82b63383c52ef874e941a280c0/loadbj.m#L367-L370

[nebkat]

@fangq right, had that referenced somewhere originally but after many edits it got lost!

If we did want to allow it to appear anywhere, we would treat [$][N][N][U] as [$][U], but the fact that that is explicitly forbidden even in UBJSON is what made me assume that it was meant to still be treated as a real value - just one that doesn't end up in any containers.

I figure in practice it is almost never used, and where it is used it is almost exclusively in un-delimited sequences or arrays. All the other possible locations do not really provide any benefit because they would be immediately following a very short sequence of tokens (e.g. between the S and it's length, or between the [$U and its count), in which case the server should just not send them in the first place.

[fangq]

can you remind me what is the N marker support in the UBJSON/BJData decoder implementations in nlohmann/json? I haven't revisited it for a few years and can't remember what positions are allowed, what are not.

If we did want to allow it to appear anywhere

I am not against limiting its use in BJData decoder. There is only one other place I had used N

https://github.com/NeuroJSON/jsonmmap/blob/main/JSON-Mmap_Specification.md#on-disk-value-replacement-with-equal-or-less-bytes

in the above JSON mmap spec, I hope to achieve in-place on-disk value update with memmap-like operations in both JSON and BJData/UBJSON, where JSON files can use white-spaces as fillers to provide extra rooms to allow in-place overwriting of values; and in BJData/UBJSON, N can be used as a filler to expand the data-byte-length without causing compatibility issues. If we limit the places where N can be inserted, this may limit what data files can be overwrite on disk.

[nebkat]

can you remind me what is the N marker support in the UBJSON/BJData decoder implementations in nlohmann/json? I haven't revisited it for a few years and can't remember what positions are allowed, what are not.

I believe this covers all possible combinations: https://godbolt.org/z/r6z13erbo

in the above JSON mmap spec, I hope to achieve in-place on-disk value update with memmap-like operations in both JSON and BJData/UBJSON, where JSON files can use white-spaces as fillers to provide extra rooms to allow in-place overwriting of values; and in BJData/UBJSON, N can be used as a filler to expand the data-byte-length without causing compatibility issues. If we limit the places where N can be inserted, this may limit what data files can be overwrite on disk.

I'll give this some more thought tomorrow, but I'm pretty sure this would remain covered if we allow it in arrays/objects/top-level streams. We would only be:

1. Restricting it is in weird places such as between $/# and S/length, which couldn't be N-ified either way.
2. Stating explicitly that [N][T] at the top level is not equivalent to [T], rather the "ignoring" effect is a function of a "stream parser". This is to counter the way nlohmann currently treats such inputs, which feels wrong to me.

In other words, it is a real value (albeit one that encodes the absence of a value), not a magic protocol level skip that can be placed anywhere.

Considering the white-spacing in JSON would not work for {"foo": "value to be removed"} without also removing the key, I think it would also be reasonable to restrict [{][i][3][foo][N][}].

[nebkat]

Finally returning to this.

I think by changing no-op from the rather ambiguous "valueless" value to simply a skip signal for containers makes it much easier to understand and doesn't affect any legitimate existing uses.

This puts it in a similar category to $ and # which are not in the types table.