Note

In this article we focus on the python implementation of CCastle Compiler, and use python terminology. Partly, because it is needed to design that well, partly because this is a first step into the ultimate goal, the CCastle implementation of all Castle Workshop Tools.
And, because “the zen of python” is a good inspiration, studding it seems wise.

Short: PyPackages (START/DRAFT)

Most of the AR-components as described in e.g ‘Architectural overview’ are plugins: independent deplorable packages. As there are two views of packages [1], associated by how to use them: (1) import and (2) distribution packages, we need to design this well.

Seen from the ‘4+1 Architectural View Model’ [2] this is part of the “Deployment view”. And as such, a valuable reason to describe & study the options in details

Summary & Conclusions (ToDo)

For those, that are not interested in de design-study, this is the result

• The top-level namespace is ‘castle’, for all public, more-or-less official packages

• Only base packages (which can’t be a plugin) are located directly in ‘castle’. Also some generic “main/app” routines can be located there.

  • castle.aigr, the common intermediate langage

  • castle.monorail, the code to load the plugins and put them in the pipeline

• Most packages are plugins and use a namespace that denotes the “location” in the pipeline; in plural form to denotes the are options.

  • castle.readers.<name>

  • castle.transformers.<name>

  • castle.writers.<name> (not backends, as only the writer-part is part of Castle.)

• Use “pep 420” native namespace packages for all plugins.

  • In short: plugins should never put an __init__.py file in a shared namespace-part (like castle.readers.)

  • Only the “main” package is allowed to put files there.

• Optional, auxiliary (plugin) packages are usually named (placed-in)

  • castle.<loc>.plugins.<opt>.

• Local (non public) extensions should normally not be in ‘castle’.

• Optional plugins, that can be shared by many plugins for test, debug & development purposed are allowed in one place in the castle-hierarchy,

  • castle.TESTDOUBLES.<hier>,

    • TESTDOUBLES is in capitals, to signal it’s a specical case

    • Those plugins should not have production code – no application should depend/need those
      (But for test & verification)

    • ‘<hier>’ is typical namespace, below castle, like: aigr

  • castle.TESTDOUBLES.aigr.sieve holds (parts of) ‘The Sieve (basic variant)’ in ‘Abstract Intermediate Graph Representation’, As test-input for e.g. plugins. Or as reference for a reader.

Hint

Suppose we make the “nice-fsm-plugin” that act as transformer …

Note

The resulting namespace will be:
castle.transformers.fsm.nice_fsm

The name of the top directly does not matter (for python). We advice a name that shows it’s not a python-id, and is related to the package name. For example:
<nice-fsm-plugin>

This directory contains the only package-info-file, which name a content depends on the (package) build systems. E.g.

• pyproject.toml (setup.py is outdated)

  • name :str e.g. castle-RPy-writer or “nice-fsm-castle-plugin”

  • version :str typical a dotted number (as string)

  • dependencies :List[str] =[ castle-aigr, <package-names>, …]

We need a file-hierarchy, with empty directories, that mirror the (shared) namespace

• <nice-fsm-plugin>/castle/ – empty, no __init__,py

• <nice-fsm-plugin>/castle/transformers/ – empty, no __init__,py

• <nice-fsm-plugin>/castle/transformers/fsm – empty, no __init__,py

• <nice-fsm-plugin>/castle/transformers/fsm/nice_fsm Holds all code:

  • __init__.py can be empty, but typically uses from .<> import <> to have the “api functionality” available

  • <files>.py code

  • <sub>/ sub-package, (full-dirname to be included in packages=[…] above

Opportunity

Let study the options, our needs and how others handle packages & plugins, before designing for castle.

Importing

When importing a package we (typical) use a hierarchical namespace to make the functionality of (a part of) the package available. For example import castle.aigr will load that common package.
It dotted name show aigr is part of castle.

Packages that are optional, or where alternatives are available, are conveniently bundled in an extra ‘layer’:

• from castle.readers import typicalReader as reader

• from castle.TESTDOUBLES.readers import mockReader as reader (selected option for mocks)

• from castle.readers import mockReader as reader (alternative, not preferred)

The “dotted names” gives the user/SW-engineer an hint on which (sub)packages are available, and where it fits. Aside of that, the name is not very important. During importing we can even rename a package, with the as <name> langage feature.

Note

The functionality does not depend on the name!

After a package is imported, all it’s functionality is available; always. The name of the (sub)package isn’t relevant, nor is the (disk) location. It is also possible to use castle-plugins when they are not located in the castle.namespace. As shown by the next example:
from myLocal.Hack import MyOwnReader as reader (not advised).

Installing

When (pip) installing a distribution package, typically a zip-file is downloaded, and extracte into a directory. The name of that directory typically correspondes with the namespace, when importing.
The developer of a distribution specifies (has to design) in which namespaces (“directory”) the functionality becomes available.

For small, simple packages this is straightforward. Typically, the directories ‘in’ the distribution (zipfile) match the name(s) that are used to import it.
For bigger packages –especially when using plugins, and multiple parties can contribute and (independently) distribute sub-packages– it a bit more complicated. All parties should agree on the name(s), to make is clear for the users.

A typical user expect that all packages for CCastle become available in castle., or something that is close to it [3]

Related

There is a clear relation between the tow kind of packages, and for small packages the are typical 1:1. All (importable) sub-packages are in the same “zip.
For bigger packages/distributions – and certainly for when using “plugins”– this is not the case.

Namespaces Packages

It is possible to combine several distribution packages into one importable packages (structure). This is called namespace package(s). Since python-3.3 (pep 420), this is standardised in Native namespace packages.

In short: each independent sub-package-developer should use of a common (same name), but “empty” top-directory. That dir should be ‘in’ the distribution-package (aka the zipfile). That dir should only contain a (1) sub-dir. And all files should be in that sub-dir.

Warning

The name of that common directory should be aligned!

XXXX

Why?

Clarity

It should be easy for the user to understand which package is related to CCastle and how it is related. Some packages are mandatory (e.g castle.aigr) or are needed in de base-setup, others are (external) plugins. But packages also have a place in the AIGR pipeline – it’s convenient to effortlessly tell readers and backends (etc) aside.

Ownership

Each (distribution) package has an owner too. Typical, the have there own “code archive”, own “package-numbering” and “release cycle”, ect. This also aplies to plugins! They should be (able to) deployed independently.
In practice, “ownership” (like: who owns the code-archive, but also “when” to release) are the borders that define the distribution-packages.

---

XXXX

---

Footnotes

[1]

See also

https://packaging.python.org/en/latest/tutorials/packaging-projects

[2]

I often use a “modern, OO variation of the standard 4+1 AR model” to describe architecture. As the original paper comes from the pre-OO-era, its jargon is a bit outdated.
See e.g. http://systemenbetermaken.nl/AR/ for some slides about it – one-day I will make a blog ..

[3]

For example many extensions for “Sphinx” are available as a sub-package of sphinxcontrib, where sphinx itself is locates in sphinx (notice: sphinxcontrib is used, not sphinx.contrib – possible because namespace-package wheren’t available/standardised when sphinx(contrib) started.)
In this location one can find mostly “standard extensions”.

Other (independent) sphinx-additions are installed (aka can be found) in sphinx_<name>. Despite those namespaces (and directories) are completely independent for a human/user it clear the are related.

Comments

comments powered by Disqus