Skip to content

NEP-0101: NOC Packaging

This package describes an approach to NOC packaging, distributing, and delivering.

Addressed Problem

Historically, the NOC source code has been placed in the root of the repository. For example, the noc.aaa module resides in the aaa/ directory. This layout dates back to a time when Python’s packaging ecosystem was still immature and best practices were not yet established.

As a result, building proper Python packages for NOC was never prioritized or fully implemented. Instead, NOC has traditionally been installed by cloning the repository—first via SVN, then Mercurial, and now Git. Even with the advent of Docker and containerization, this approach has persisted.

This situation has led to a number of philosophical, technical, and legal issues, which are detailed below.

Legal issues: 1. No tangible distribution artifact: The NOC distribution is merely a pointer to the repository. There is no physical, measurable object that constitutes a NOC installation. This ambiguity complicates agreements between providers and consumers and can lead to serious misinterpretations, including license violations. 2. Source code exposure: The source code is delivered as part of the installation process, effectively granting customers the right to modify it — intentionally or not.

Technical Issues 1. Git required for deployment: The repository is used not just for development, but also for deployment—an anti-pattern that blurs responsibilities. 2. Immutability issues: Because Git is tied to deployment, rewriting commit history becomes impossible. 3. Infrastructure lock-in: This tight coupling makes NOC dependent on specific Git hosting platforms like GitLab or GitHub. 4. Lack of separation: There is no clear distinction between what lives in the repository and what should actually be delivered. This conflicts with a monorepo philosophy. 5. Implicit namespace: The noc namespace is never an actual directory on disk. To make Python’s import mechanism work, a PYTHONPATH hack is needed:

  • Clone repo into noc directory.

  • Set PYTHONPATH to ..

  • Fixed installation path: As a result, NOC must always be installed in a directory named noc (typically /opt/noc), which often frustrates system administrators.

  • Tooling issues: Linters, IDEs, and other tools struggle with the implicit namespace. This degrades code quality and hampers navigation.
  • Fragile dependency management: Dependencies are specified in .requirements/ files, which must be carefully handled during deployment.
  • Incompatibility with setuptools: The current layout is incompatible with standard Python packaging tools like setuptools, which expect code to live under noc/ or src/noc/.

Root Cause

The core issue is the absence of a clear contract between development and deployment. These concerns are conflated, making both processes fragile and error-prone.

Proposed Solution

The traditional and effective solution is to introduce a packaging stage between development and deployment. This cleanly separates the development environment from the production environment.

A widely accepted best practice is to place all Python code under a src/noc/ directory. This structure is supported out of the box by:

  • setuptools
  • Linters
  • IDEs

Migration Challenge

However, migrating to this layout creates a chicken-and-egg problem:

  • Moving the source code breaks all existing deployments.
  • It also leaves pending merge requests in an unresolved state.

Proposed Solution

Step 1: Introduce a Symlinked src/noc Layout

Create a src/noc directory and populate it with symlinks pointing to the actual source files and directories that are intended for delivery.

The expected layout should look like this:

lrwxrwxrwx@ 1 dv  staff  17 Apr 22 10:26 __init__.py -> ../../__init__.py
lrwxrwxrwx@ 1 dv  staff   9 Apr 22 10:26 aaa -> ../../aaa
lrwxrwxrwx@ 1 dv  staff   8 Apr 22 10:26 bi -> ../../bi
lrwxrwxrwx@ 1 dv  staff   8 Apr 22 10:26 cm -> ../../cm

This change is non-invasive:

  • Existing and future deployments remain fully functional, as the actual source files are not relocated.
  • Pending merge requests are unaffected, since the original file structure stays intact.

In essence, we are simply introducing an additional directory (src/noc) that mirrors the necessary modules via symbolic links, laying the groundwork for a smoother packaging process without disrupting ongoing development or deployment workflows.

Step 2: Update PYTHONPATH and Developer Workflow

Developers must update their environment settings by replacing:

PYTHONPATH=..

with

PYTHONPATH=src

From this point forward:

  • IDEs and linters will recognize the noc namespace properly, enabling accurate code navigation, linting, and auto-completion.
  • All development work should be focused on the files inside the src/noc directory.
  • Developers are no longer required to clone the repository into a noc-named directory. Any location will work, as long as PYTHONPATH=src is set.

This step significantly simplifies the development setup and improves compatibility with modern Python tooling.

Step 3: Replace open() with importlib.resources

After introducing packaging, we can no longer rely on using open() to read files and templates by relative filesystem paths. Packaging tools like setuptools may include files inside wheels or zipped distributions, making direct file access unreliable or impossible.

All such occurrences must be refactored to use the importlib.resources module, which provides a reliable and portable way to access data files bundled with Python packages.

Before (unsafe after packaging): 

with open("templates/report.html") as f:
    template = f.read()

After (packaging-friendly):

from importlib.resources import files

template = files("noc.templates").joinpath("report.html").read_text()

This ensures compatibility with packaged distributions and is the recommended modern approach for working with static assets, templates, and other resource files.

Step 4: Add Packaging Configuration to pyproject.toml

The first step of the actual packaging process is to define the [project] section in pyproject.toml, along with the appropriate [tool.setuptools] configuration.

This configuration describes the package metadata, dependencies, and layout, and is required to build standard-compliant Python distributions.

With this configuration in place, you can build a proper Python wheel for NOC using standard tools like:

python -m build

The resulting .whl file can then be published to an internal or public PyPI repository, allowing NOC to be installed via:

pip install noc

This marks the transition from source-based deployment to a modern, package-based workflow.

Step 5: Define noc Entrypoint for CLI Access

Add a console script entry point for noc in the pyproject.toml file. This ensures that noc is installed as a system-wide command, making it accessible from anywhere in the environment.

Outcome:

  • The noc command becomes globally available after installation.
  • All existing script and deployment scenarios that rely on ./noc must be updated to use just noc instead.

Step 6: Update Deployment Playbooks

All deployment playbooks must be updated to replace the git clone step with installation of the NOC Python package.

Instead of cloning the repository and running from source, NOC will now be installed via pip, using a standard Python packaging workflow.

Closed Network Support:

To support deployments in air-gapped or closed network environments, the package distribution system can include a local PyPI-compatible index, served by a lightweight tool like tower or pypiserver.

This satisfies security and compliance requirements while enabling flexible and reproducible deployments.

Step 7: Create a Legacy Mirror for Existing Installations

To preserve stability for current installations that rely on source-based deployments, a snapshot of the current repository must be cloned to a separate location—referred to as the legacy mirror.

This mirror will act as a frozen, stable source for systems that are not yet ready or intended to upgrade to the new package-based workflow.

Purpose:

  • Prevents breakage for stable environments not yet migrated.
  • Ensures long-term maintainability by decoupling legacy systems from active development.
  • Establishes a clean line between old and new deployment strategies.

This step finalizes the transition to a dual-mode support model: legacy (frozen Git) and modern (packaged NOC).

Step 8: Rewrite Git History to Move Code to src/noc

In this final step, the Git history must be rewritten to move all source files from the repository root into the src/noc directory. This change will permanently align the repository structure with modern Python packaging practices.

What Will Happen

  • All relevant Python modules and packages will be relocated under src/noc/ in all commits, not just the latest one.
  • This operation will break all existing Git mirrors and developer clones due to the changed commit hashes.

Justification

  • New installations will not be affected, as they rely on the packaged distribution (wheel).
  • Existing deployments are protected by the legacy mirror established in Step 7.
  • The result is a clean, modern, and consistent repository layout, compatible with tools like setuptools, linters, IDEs, and static analyzers.

Risk Mitigation

The damage surface is:

  • Known: Only Git clones and mirrors will be affected.
  • Controllable: All developers and mirror maintainers can be notified in advance.
  • Temporary: Developers can re-clone the repository or reset their remotes following provided instructions.

With proper coordination, clear documentation, and staged rollout, this critical step brings long-term benefits to the project’s maintainability and tooling compatibility.

Conclusion

The steps outlined in this process guide NOC through a crucial transition from a legacy Git-based deployment model to a modern, package-based distribution. By introducing packaging, symlinks, and proper configuration, NOC becomes more maintainable, deployable, and compatible with modern tooling, including IDEs, linters, and package managers like pip.

Key outcomes of this migration include:

  • Improved developer experience through proper namespace management and packaging standards.
  • Streamlined deployment workflows via Python wheels, enabling easier installation and version management.
  • Preservation of existing deployments with a legacy mirror, ensuring that no current installations are disrupted.
  • A clear path forward for future enhancements and upgrades, as the project now adheres to established Python packaging practices.

While the migration requires significant changes, especially in Git history and deployment playbooks, the benefits of a cleaner, more flexible codebase and deployment process far outweigh the short-term disruptions. By following these steps and maintaining clear communication with developers and stakeholders, NOC will be positioned for long-term success and stability.