Short: PyPackages (START/DRAFT)

Most of the AR-components as described in e.g ‘Architectural overview’ are plugins: independent deplorable packages. As there are two views of packages [1], associated by how to use them: (1) import and (2) distribution packages, we need to design this well.

Seen from the ‘4+1 Architectural View Model[2] this is part of the “Deployment view”. And as such, a valuable reason to describe & study the options in details

Summary & Conclusions (ToDo)

For those, that are not interested in de design-study, this is the result

  • The top-level namespace is ‘castle’, for all public, more-or-less official packages

  • Only base packages (which can’t be a plugin) are located directly in ‘castle’. Also some generic “main/app” routines can be located there.

    • castle.aigr, the common intermediate langage

    • castle.monorail, the code to load the plugins and put them in the pipeline

  • Most packages are plugins and use a namespace that denotes the “location” in the pipeline; in plural form to denotes the are options.

    • castle.readers.<name>

    • castle.transformers.<name>

    • castle.writers.<name> (not backends, as only the writer-part is part of Castle.)

  • Use “pep 420” native namespace packages for all plugins.

    • In short: plugins should never put an file in a shared namespace-part (like castle.readers.)

    • Only the “main” package is allowed to put files there.

  • Optional, auxiliary (plugin) packages are usually named (placed-in)

    • castle.<loc>.plugins.<opt>.

  • Local (non public) extensions should normally not be in ‘castle’.

  • Optional plugins, that can be shared by many plugins for test, debug & development purposed are allowed in one place in the castle-hierarchy,

    • castle.TESTDOUBLES.<hier>,

      • TESTDOUBLES is in capitals, to signal it’s a specical case

      • Those plugins should not have production code – no application should depend/need those
        (But for test & verification)

      • ‘<hier>’ is typical namespace, below castle, like: aigr

    • castle.TESTDOUBLES.aigr.sieve holds (parts of) ‘The Sieve (basic variant)’ in ‘Abstract Intermediate Graph Representation’, As test-input for e.g. plugins. Or as reference for a reader.


Suppose we make the “nice-fsm-plugin” that act as transformer …


The resulting namespace will be:

The name of the top directly does not matter (for python). We advice a name that shows it’s not a python-id, and is related to the package name. For example:

This directory contains the only package-info-file, which name a content depends on the (package) build systems. E.g.

  • pyproject.toml ( is outdated)

    • name :str e.g. castle-RPy-writer or “nice-fsm-castle-plugin”

    • version :str typical a dotted number (as string)

    • dependencies :List[str] =[ castle-aigr, <package-names>, …]

We need a file-hierarchy, with empty directories, that mirror the (shared) namespace

  • <nice-fsm-plugin>/castle/ – empty, no __init__,py

  • <nice-fsm-plugin>/castle/transformers/ – empty, no __init__,py

  • <nice-fsm-plugin>/castle/transformers/fsm – empty, no __init__,py

  • <nice-fsm-plugin>/castle/transformers/fsm/nice_fsm Holds all code:

    • can be empty, but typically uses from .<> import <> to have the “api functionality” available

    • <files>.py code

    • <sub>/ sub-package, (full-dirname to be included in packages=[…] above


Let study the options, our needs and how others handle packages & plugins, before designing for castle.


When importing a package we (typical) use a hierarchical namespace to make the functionality of (a part of) the package available. For example import castle.aigr will load that common package.
It dotted name show aigr is part of castle.

Packages that are optional, or where alternatives are available, are conveniently bundled in an extra ‘layer’:

  • from castle.readers import typicalReader as reader

  • from castle.TESTDOUBLES.readers import mockReader as reader (selected option for mocks)

  • from castle.readers import mockReader as reader (alternative, not preferred)

The “dotted names” gives the user/SW-engineer an hint on which (sub)packages are available, and where it fits. Aside of that, the name is not very important. During importing we can even rename a package, with the as <name> langage feature.


The functionality does not depend on the name!

After a package is imported, all it’s functionality is available; always. The name of the (sub)package isn’t relevant, nor is the (disk) location. It is also possible to use castle-plugins when they are not located in the castle.namespace. As shown by the next example:
from myLocal.Hack import MyOwnReader as reader (not advised).


When (pip) installing a distribution package, typically a zip-file is downloaded, and extracte into a directory. The name of that directory typically correspondes with the namespace, when importing.
The developer of a distribution specifies (has to design) in which namespaces (“directory”) the functionality becomes available.

For small, simple packages this is straightforward. Typically, the directories ‘in’ the distribution (zipfile) match the name(s) that are used to import it.
For bigger packages –especially when using plugins, and multiple parties can contribute and (independently) distribute sub-packages– it a bit more complicated. All parties should agree on the name(s), to make is clear for the users.

A typical user expect that all packages for CCastle become available in castle., or something that is close to it [3]

Namespaces Packages

It is possible to combine several distribution packages into one importable packages (structure). This is called namespace package(s). Since python-3.3 (pep 420), this is standardised in Native namespace packages.

In short: each independent sub-package-developer should use of a common (same name), but “empty” top-directory. That dir should be ‘in’ the distribution-package (aka the zipfile). That dir should only contain a (1) sub-dir. And all files should be in that sub-dir.


The name of that common directory should be aligned!




It should be easy for the user to understand which package is related to CCastle and how it is related. Some packages are mandatory (e.g castle.aigr) or are needed in de base-setup, others are (external) plugins. But packages also have a place in the AIGR pipeline – it’s convenient to effortlessly tell readers and backends (etc) aside.


Each (distribution) package has an owner too. Typical, the have there own “code archive”, own “package-numbering” and “release cycle”, ect. This also aplies to plugins! They should be (able to) deployed independently.
In practice, “ownership” (like: who owns the code-archive, but also “when” to release) are the borders that define the distribution-packages.




comments powered by Disqus