stdlib: adds itertools, composable templates for inline iterators

[ZoomRmc]

This PR implements RFC #562 , it was given a green light long ago.

The code is based on beef's idea and impl in slicerator/itermacros.

For convenient syntax that allows writing "foo".split.mapIt(it).collect() and not mapIt("foo".split, it).collect()" it requires merging #25679 (or an alternative fix) but it doesn't rely on it inherently (except some examples and test cases assume this form being available), so workable even without it.

This example works now:

import itertools
import std/[strutils, options]

# Find the first element in a sequence of transformed numbers above 35.
# Note using `Slice[int].items` instead of `CountUp` (also supported).
doAssert (-25..25).items.mapIt(it * 10 div 7).findIt(it > 35) == none(int)

# Filter philosophers by country, compose sentences, join to a string.
let philosophers = {
  "Plato": "Greece", "Aristotle": "Greece", "Socrates": "Greece",
  "Confucius": "China", "Descartes": "France"}

const Phrase = "$1 is a famous philosopher from $2."
let facts = philosophers.items()
              .filterIt(it[1] != "Greece")
              .mapIt([it[0], it[1]])
              .mapIt(Phrase % it)
              .foldIt("", acc & it & '\n')
doAssert facts == """
Confucius is a famous philosopher from China.
Descartes is a famous philosopher from France.
"""

The implementation is straightforward, API is bog-standard and shouldn't be surprising to anyone familiar with this style, tests cover everything. The only interesting thing in the code is that it uses concepts for overloads without any when compiles hacks, feels pretty great.

The set of provided templates is basic and the naming is all common, so I hope there's not much place for bikeshedding. I've been craving for this feature since 2018 or so and didn't hear much pushback against the idea, so I hope this PR doesn't go to waste.

requires #25679

[ZoomRmc]

Test failures all should get green after #25679.

[arnetheduck]

Very nice, specially that things compose and don't have to be materialized!

1. Can closure iterators be supported as well out of the box via some standard helper? feels like a pretty sad omission
2. does the whole thing evaluate to an iterator itself, ie does for x in (-25..25).items.mapIt(it * 10 div 7): echo x work?

[ZoomRmc]

Thanks!

1. Can closure iterators be supported as well out of the box via some standard helper? feels like a pretty sad omission

Well, it's a language omission first (iterator (): T isnot iterable[T]) and I don't think it doable now without instantiating an inline iterator that uses the closure first.

This works:

  let infinite = iterator(): int =
     while true: yield 1

  template asIterable[T](it: iterator(): T {.closure.}): untyped =
    iterator wrap(): T =
      for x in it():
        yield x
    wrap()

  var sum = 0
  for i in infinite.asIterable().take(3):
    sum.inc i

  doAssert sum == 3

We could provide such template in the module, but it's a hack and we should just make iterator {.closure.} satisfy iterable[T].

2. does the whole thing evaluate to an iterator itself, ie  does `for x in (-25..25).items.mapIt(it * 10 div 7): echo x` work?

Absolutely! Otherwise, the templates wouldn't compose. Added a test case for the chain in the regular iterator context (the for loop).

[ZoomRmc]

CI errors are unrelated timeouts now.

[AmjadHD]

Why is when defined(nimdoc) needed?

[AmjadHD]

iterator (): T isnot iterable[T]

Araq suggested unifying closure and inline iterators.

[ZoomRmc]

Araq suggested unifying closure and inline iterators.

I don't know how that would look like, because they are inherently different. Inlines are just a syntactic sugar, but closure iterators are 1. callable 2. have state 3. enclose around stuff.

This entails some serious design work.

However, just making them satisfy iterable[T] (which is template-only faux-type) can go a long way.

Why is when defined(nimdoc) needed?

Adding doc comments to templates producing untyped nodes results in wrapping a call in a statement list and, as a result, type mismatch after expansion.

import std/macros

macro genIter(): untyped =
  quote do:
    iterator tmp(): int = yield 42
    tmp()

template clean(): untyped =
  genIter()

template dirty(): untyped =
  ## A flower?
  genIter()

macro dump(): untyped =
  echo "CLEAN"
  echo treeRepr getAst clean()

  echo "\nDIRTY"
  echo treeRepr getAst dirty()

dump()

produces

CLEAN
Call
  Sym "genIter"

DIRTY
StmtList
  Empty
  Call
    Sym "genIter"

[AmjadHD]

Can you check if it's a stmt list, and unwrap it if so?

[ZoomRmc]

Can you check if it's a stmt list, and unwrap it if so?

Of course, but only from a macro. You can't operate on nodes from a template.