Tip

QuickNotes are short, quick, informal notes. Often to order my thoughts and analyse the options before I write the “real” design (or other docs).
They are unofficially published. You can read them but do not expect that they are well written. Or till actually, as maintaining them is not a goal.

QuickNote: The Sieve in RPython

Before implementing an RPython backend, a short study of RPython is made, including implementing The Sieve (basic variant) in RPython. It’s like a hand-compiled RPython variant, following the approach as described in the Jinja rendering study), for the CC2CPy backend.

Writing RPython isn’t too complicated for someone with a C-background. But making the code pass the RPython compiler can be tricky, however. Both, as just added statements can interfere with existing code – when that isn’t “static enough”, new code can trigger compile errors in old, tested code! And, because the RPython compiler isn’t that great in giving helpful info.
It feels like old-style debugging: try-and-error, remove-and-enable-lines, one-at-a-time until one understands and it becomes easy to fix…

This blog is both to share some RPython experience and to study the patterns that are needed to generate RPython code to implement the Castle-Compiler.

Code

Below we show only a part of the (manually written) code. It roughly follows the “handCompiled-C” version (see CC2Cpy: CCastle to C compiler in Python). Again, mostly the structure to get the (Castle) Components working, like sending events between Elements is presented.
All other code is quite straightforward, and can be found in the code-archive

For this study, I had two objectives: (a) make it (functional) work in RPython, and (b) find the patterns to automate. So, do not expect greatly crafted (maintainable) code.

Definitions

In Castle, you use Events, Protocols and Components (to name a few), all will result in some data structures [1] that define them: build-in classes that are (statically) instantiated.
This part is easy to automate.

CC_B_ProtocolCC_B_ComponentInterfaceCC_B_ComponentClass

StartSieveSimpleSieveSlowStart (XXX)

CC_P_StartSieve_runTo    = 7       # XXX Now arbitrary number, simulating 7 [0:6] inherited events
CC_P_StartSieve_NoEvents = 8       # XXX This number is ``CC_P_StartSieve_runTo+1``

cc_P_StartSieve = buildin.CC_B_Protocol(name = "SimpleSieve",
                                            kind = buildin.CC_B_ProtocolKindIs_Event,
                                            inherit_from = None,
                                            events = [])
cc_P_StartSieve.events.append(
    buildin.CC_B_P_EventID(name="runTo",
                               seqNo=CC_P_StartSieve_runTo,
                               part_of=cc_P_StartSieve))
cc_P_StartSieve.events.append(
    buildin.CC_B_P_EventID(name="newMax",
                               seqNo=CC_P_StartSieve_NoEvents,
                               part_of=cc_P_StartSieve))

Hint

Here,the basic cc_P_<<ProtoName>> structure is defined first, without events. Then, the events are added, one by one; by appending to the (internal) list.

As those CC_B_P_EventID(s) refer the Protocol-structure, there is no syntax to combine that (now).
You will see this approach also in other definition.

Note

It’s possible to improve this code in many ways. For example, we can use smart initialisers to get define a data-structure in ones.
However, as it is (a) an experiment and (b, more relevant) it’s (ultimately) generated code, that isn’t needed, nor foreseen.

Data structures

Each (Castle) component results in a (RPython) class containing all code for that component. As (R)Python classes are also namespaces, the names of (e.g.) the even-handlers become shorter; there is no need to put the component-name in the method-name. But in general, we follow the structure as in the CC2Cpy approach.

Components

A (Castle) Component becomes a subclass of CC_B_Component, an (abstract) base class providing (currently mostly) debugging support. Subclasses should override the _debug_attr_() method, which returns (but not print) a string with name=value pairs for debug-printing.

The initialiser is split in __init__() and _castle_init(); the latter implements (generates code for) the Castle-init function, and is called by __init__() [2].

Both initialisers will have the parameters to the class, and/or the elements. A default value (Nil) is set in __init__, whereas _castle_init can set it – as specified in the castle-code. See the SieveClass.py code for an example (not shown).
Static variables in Components –which are defined in e.g. EventHandlers– are also handled in __init__. As they have a scope of one function, their name is prefixed with that function-name. You can find an example in the GeneratorClass.py (not shown).

Roughly each protocol-event for each port (aka an EventHandler) results in a (R)Python method, as do data- (and other) handlers, and internal functions.

DispatchTables

Every (input) port can trigger several EventHandlers (or data-/stream-/… handlers), as an (event) protocol can have multiple events. Therefore, a component has an (event) DispatchTable per port [3].

Every DispatchTable is an array (python: List) with the same size as the number of events in the corresponding Protocol (which is static at compile-time) – including the inherited events [4].

EventHandlers: 2 kind (syntax demo)

interconnectsub-elements

SimpleSieve.input(try) on self.try
{ ... }

Pins

A (Castle) component can have sub-components. They can be interconnected using the typical syntax and (RPython) structure.
But a component-element can also receive and send messages to its sub-elements. The (SieveDemo) Main component uses this to handle a newly found prime (by the Finder’s output port found). In Castle-code the EventHandler names both the (sub)element and its port (instead of only a port). See the sidebar for an example.

Such an “incoming event” of a sub-element needs a DispatchTable also, we call them “pin(s)”.
Note: No extra DispatchTables are needed to send an event to a sub-element; the normal in-port of that component (aka it DispatchTable) handles that. A bit of extra gear is needed to fire that event, however (see below).

Examples

CC_B_ComponentDispatchTable(s)

GeneratorSieveFinderMainBoard (inherited)

class CC_Generator(buildin.CC_B_Component): # generated class

   # A few standard method
   def __init__(self,  *args): ...
   def _castle_init(self): ...
   def _debug_attr_(self, name_only=True): ...

   # One method for each event-port combi
   def StartSieve_newMax__controll(self, max_): ...
   def StartSieve_runTo__controll(self, max_): ...
   def SimpleSieve_input__collect(self, foundPrime): ...

   # other
   def assertCorrectPrime(self, foundPrime): # internal, test (manually written)

Code fragments

To connect two components (via ports), and to send events (or other protocols) over them, some code is needed. That is studied below.

Connecting

Castle supports multiple Machineries, with all will generate a bit different code. Here we use the DirectCall one [6].

We use (one of the ) same examples as in QuickNote: Connecting two ports (see: CC_Mi_Main__init).
The Castle Code:

.generator.outlet = .finder.newPrime;

This will result in two lines of RPython. They are very similar to the C-backend: the outlet field holds the other component (element) and the handlers denote the DispatchTable of the incoming port.

self.generator.outlet.connection = self.finder
self.generator.outlet.handlers = FinderClass.cc_S_Finder_newPrime

Sending

Again, we use an earlier example (in QuickNote: Sending Events & Machinery). The Generator is sending numbers to the sieves

self.outlet.input(i); // Part of Generator:: StartSieve.runTo(max) on self.controll

This translates to a few lines of Python.
The manual crafted version adds some print/debug and assert statements. It also handles the case where a None is set in a DispatchTable. See the real code for the details. Here you see the essence.

outport  = self.outlet
receiver = outport.connection
handlers = outport.handlers
handler = handlers[SimpleSieve.CC_P_SimpleSieve_input]
handler(receiver, i)

Or simply:

outport  = self.outlet
index    = SimpleSieve.CC_P_SimpleSieve_input
handler  = outport.handlers[index]
handler(outport.connection, i)

Note

This code assumes there is always a valid method in all entries of the DispatchTable. See the generic error function footnote.

Trigger sub-elements

To send an event to an (in)-port of a (Castle) sub-element, we need to select the correct (RPython) DispatchTable, index it with the event number, and call that method, passing the sub-element’s (RPython) instance – and parameters.

GeneratorClass.cc_S_Generator_controll[StartSieve.CC_P_StartSieve_runTo](self.generator, max)

Warning

The Castle Code for this feature isn’t 100% clear. Several variants of the (Castle) Sieve do it differently, wrong or simply lack that line [7].

---

See also

• https://osdn.net/users/albertmietus/pf/TryOut_PyPy-and-Rpython (SLOW!)
  My collection of small rpython demos, and typical Python-code that is not RPython ..

• https://osdn.net/users/albertmietus/pf/CC_Sieve_in_rPython (SLOW)
  The code used for in this study.

Footnotes

[1]

To implement them dataclasses would be natural. They are, however, introduced in Python-3.7. As RPython is a Python-2.7 subset, they are not available.

[2]

Possibly, it would be better to have 3 init functions. Now the (generated) __init__, calls both the __init__ of the base-component/class and its own _castle_init.
This results that all Caste-init code is called, mixed with build-in/generated init code.

For now, this approach is used. Remember: the goal in generated code; with different demands on maintainability.

[3]

We could implement this by making the ports “active”; an RPython class with implements a method for each event – then we use the DispatchTable of Python itself. However, that port method needs to call a method in the ComponentClass – whose name is not statically determined.
In (normal) Python, this is possible using introspection. That feature is however not allowed in RPython.

As all code is generated, it is still possible. We could generate a port method that calls a component method, as we know the name of the latter during (Caste) compile-time. But is that better?
At the moment, we believe a table approach, doing the same thing directly is better!

[4]

In this example, we use some arbitrary number (like 6, 7) of “dummy” inherited events for all Protocols. Those not existing inherited ones get a simple None in the table. In real code, this should not happen. Then (a) inherited events are copied down, and (b) empty slots will contain the generic error function [5].

[5]

The runtime becomes simpler (and faster) by using an error function, then using None, and checking for None on every call. For this experiment, both are not needed.

[6]

As we have seen in DirectCall, the code for other machinery differ, but typically not too much.

[7]

During this study, we found several (small) bugs in the Castle Code: typos, wrong names, and (like this) lacking lines.
Probably, that is the result of writing code without a compile/syntax-checker (and so, no options for UnitTesting and/or practising TDD)

QuickNote: Connecting two ports   QuickNote: Jinja Templating for rPY

Comments

comments powered by Disqus