[eDSL] Redesign the Standard Library

[rachitnigam]

The calyx-py standard currently provides very simple, primitive wrapper over AST construction. For example, the comb_mem_d1 primitive is simply:

class Stdlib:
     ...
    @staticmethod
    def comb_mem_d1(bitwidth: int, size: int, idx_size: int):
        return CompInst("comb_mem_d1", [bitwidth, size, idx_size])

My proposal is that we take inspiration from calyx.builder and design a new interface to the standard library where users can call nice, high-level methods to perform actions with library components. For example, this is the code we should be able to write:

import calyx
from calyx.stdlib import Memory, Mult, Reg, Add

a = Memory.comb(32, 16) # Create a new combinational memory
b = Memory.comb(32, 16)
mul = Mult.seq() # A new sequential, non-pipelined multiplier

// For indexing logic
idx = Reg()
add = Add()

# b[i] = a[i] * 10
with calyx.repeat(16):
  (a_out, a_done) = a.read(idx.out) # Memory provides high-level .read(idx) method
  (prod, prod_done) = mul.apply(a_out, 10, guard=a_done)
  b_done = b.write(prod, guard=prod_done) # optional argument defines how to guard the write_en signal
  
  (add_out, add_done) = add.apply(idx.out, 1)
  idx_done = idx.write(add_out)

To make this is happen, there is a bunch of black-magic that needs to be handled in the background. Namely, each of these high-level operators like .read, .write, and .apply need to generate groups. The code above needs to implicitly sequence those generated groups to express the control program correctly. Let's break down the challenges into a couple of steps:

Rich Primitive Interfaces

Borrowing ideas from ControlBuilder and the likes, we want to provide class definitions for primitives that have methods like read and apply on them. This should be relatively straightforward: given some set of ports, generate a group and return the right ports for performing the remaining computation:

class CombMemory(Memory):
  def read(addr: Port, comp: CompBuilder):
    with comp.group(f"read_{self.name}"):
      self.addr = addr
      self.read_en = 1
    return (self.out, self.read_done)

Note that we inherit a base class Memory. The idea is that all memories inherit this class and provide a uniform interface. For example, it should be easy to:

• Swap between CombMem and SeqMem (although maybe we don't want this because of problems outlined in Deprecate std_mem #1261)
• Swap between "true" multidimensional memories and "flat" memories that automatically transform the indexing logic.

Rich Control Builder

The next step, which I confess I don't have a full technical picture of, is designing new control operators like repeat and use them to provide even higher-level operators like calyx.for which automatically generating the indexing logic.

The end-goal here is providing the ability to write high-level code to make Python DSLs more maintainable and usable.

[rachitnigam]

Tagging @sampsyo, @anshumanmohan, and @nathanielnrn for comments!

[EclecticGriffin]

Seems like a cool idea!

If we're just considering the sequential composition of groups, we could attempt to do something similar to how the GroupBuilder works. I think the challenge is that we'll need to be more clever with the definitions of the group generating methods you've described as they would need to be responsible for adding themselves to the sequential composition, since we cannot rely on clever __setattr__ implementations with the desired code. That would require further use of the thread stack stuff which may or may not be best practice. I confess that I lack familiarity with that particular point.

[anshumanmohan]

Agreed, it's a nice idea. I will just mention: as the eDSL gets Python-ier and richer, the experience of using it appears to stray further from the experience of writing straight Calyx. Eventually I can imagine having a Calyx user who can get really far using the eDSL, but then doesn't understand the .yx code that they've generated. Not sure if this is an issue!

Re: the implementation, I'm pretty shaky on the existing black magic and can't really speak to this additional black magic 😅 I'd be happy to learn though!

[sampsyo]

This is great; I like it.

One small note on the imagined syntax here:

a = Memory.comb(32, 16) # Create a new combinational memory
b = Memory.comb(32, 16)
mul = Mult.seq() # A new sequential, non-pipelined multiplier

I think we will need the initializations to know where to put the new cells. So instead of Memory.comb(32, 16), I think we want builder.comb_mem(32, 16), which creates a new Memory object tied to the builder's underlying component. Or else possibly Memory.comb(builder, 32, 16), but that is a little less nice.