Continuum Software Factory

Project as an Assembly Line

Modern enterprise projects are complex, multi-component, and multi-platform. In fact, one project may consist of 10, 20, or even 50 related sub-projects forming a dependency hierarchy between each other. Their initial setup and further maintenance often become a separate engineering effort requiring significant coordination across teams; every new change could lead to cross-component behavioral changes. Moreover, such a large structure is never static: it is a living and continuously changing system.

To treat component as a complete unit rather than just a piece of code, Continuum defines component as a set of Requirements, Architecture, Code and Tests (RACT). However, since every component is a part of a bigger system, dependencies between components must be clearly defined. Understanding and managing these dependencies is one of the critical responsibilities of a well-designed CI/CD system.

As in any factory, a greater number of checks typically leads to higher product quality. Unfortunately, there is always the tradeoff between quality and resources such as time and computing power. Running all possible validations on every change is neither practical nor scalable. The cost of poor-quality rises almost exponentially with every stage at which a bug remains undetected, making early detection critical. The only sustainable way to balance these opposing forces is through a deep understanding of system dependencies. By leveraging this knowledge, it becomes possible to execute precisely the right set of tests - targeting only the affected components, thereby maximizing defect detection where it matters most while maintaining optimal use of resources.

The exact numbers could be seen below.

Meaning every 1$ invested in saves $50-$150 downstream. For an organization spending $1M/year on release candidate incident resolution, a mature orchestration layer could reduce that number to under ~$300K.

For that reason, an orchestration layer that goes beyond simple pipeline / automation / execution was designed. Its purpose is not only to run Per-Commit, Scheduled, and Release pipelines, but to intelligently manage and coordinate them by leveraging system-wide dependencies, with the goal of detecting the majority of issues during the Presubmit and Post-Submit stages. This approach is closely aligned with the well-established shift-left testing principle, meaning that validation and defect detection are moved as early as possible in the development lifecycle, where issues are faster and cheaper to fix.

Platform Engineering: Split the Responsibility!

By definition platform engineering provides developers with secure, automated, and self-service foundations that simplify the software delivery process. By standardizing workflows for building, testing, and deploying applications across different environments, it helps teams work more efficiently while reducing the burden of operational overhead.

This "assembly line" approach does actually what platform engineering domain prescribes. It implements the idea of splitting responsibilities. So the stage logic and execution should be controlled by those who know their solution best - the project teams! Meanwhile, platform services - the processing stations - should be maintained by the Platform Engineering team.

Dynamic Pipelines: Treating Software as a Complete System

Dynamic pipelines can be understood as a Directed Acyclic Graph (DAG) of software components, where build and test dependencies are resolved hierarchically. In this model, if Component A depends on artifacts produced by Component C in order to build or execute its tests, then A is parent of C in the dependency hierarchy. An example of such a hierarchy is shown below.

This structure is defined in the human-readable configuration file and is subsequently translated into a corresponding set of pipelines.

Each component can be built for multiple platforms, and both component-level and platform-level execution are parallelized to improve throughput and reduce overall pipeline duration.

Component may have optional set of stages from the following list:

• build - Builds code for specific platform
• static_tests - Executes static analysis tasks such as linters, format checks, and similar validations
• SW_unit_tests - Runs software unit tests
• SW_integration_tests - Runs software integration tests
• SW_qa_tests - Runs end-to-end tests validating specific software requirements
• SYS_integration_tests - Runs system-level integration tests
• SYS_qa_tests - Runs end-to-end tests validating specific system requirements

V-model compliance

This approach aligns fully with the V-model when all stages are enabled, ensuring that each development activity is matched by a corresponding verification or validation step. It provides end-to-end traceability across the full engineering lifecycle, from requirements and architecture through implementation, integration, system validation, and release readiness. This makes it particularly well suited for regulated, safety-critical, and other high-assurance development environments.

At the same time, because each stage remains optional, the same framework scales naturally across very different project types. A small MVP can use only the essential build and test stages to remain fast and lightweight, while larger or more mature projects can progressively enable deeper validation, broader coverage, and stricter quality gates. In this way, the pipeline grows together with the product, evolving from a pragmatic development workflow into a complete industry-grade engineering process.

This approach also goes hand in hand with documentation-as-code, where requirements, architecture, implementation, test definitions, and test results are all managed as versioned artifacts within the same system. Test results are automatically linked to test cases, which in turn are connected to requirements, architectural elements, and implementation artifacts. As a result, the pipeline validates not only the software itself, but the consistency and traceability of the entire engineering package, significantly improving auditability, change impact analysis, and long-term maintainability.

Standardized set of pipelines

To make the platform easier to use across different projects, a standardized set of pipelines was defined to cover the majority of typical project needs.

The set is consisted of following pipelines:

• release
• reporting

Nightly and Weekly pipelines

Nightly and weekly pipelines are scheduled executions designed to validate the full state of the project at regular intervals. Unlike commit-based pipelines, they always build and test the complete set of components, providing a comprehensive snapshot of system health for a given day or week.

The primary difference between the two lies in their execution budget and test depth. Weekly pipelines typically allow for longer runtimes and therefore include broader and more detailed validation than nightly runs.

Reporting pipeline

The reporting pipeline aggregates key performance indicators (KPIs) from recent pipeline executions and publishes them to reporting portal. This provides visibility to build health, test performance, stability trends, and other operational metrics over time.

Release pipeline

The release pipeline is used to deliver a selected set of components or the entire software solution. In addition to building and validating the release candidate, it can perform several optional release management steps, including:

• Generating release notes
• Promoting artifacts
• Tagging the repositories
• Deploying artifacts to the customer environment

All of these steps are configurable and can be enabled or disabled through human-readable configuration file.

An example of release pipeline execution is shown below.

Commit pipeline

To optimize execution time for commit-triggered workflows, the orchestration layer uses the declared dependency hierarchy to determine the minimal required rebuild scope. Rather than rebuilding the entire system, it rebuilds only the components that were changed and the components that depend on them.

The following use cases illustrate this behavior.

Use case 1: Component G changed

If Component G changes, all higher-level components that depend on it must also be rebuilt and, where applicable, retested

Use case 2: Component E changed

If Component E changes, only the affected upper layers need to be rebuilt. Components in unrelated branches or deeper unaffected levels do not need to be processed.

This behavior is enabled by the orchestration layer, which determines the affected components and passes their list to the commit pipeline before triggering execution.

Real life example could be seen below:

For following pipeline configuration:

Software factory mindset

This approach allows you to treat the project build process like an assembly line: for example, an entire vehicle is assembled from large parts like the engine, wheels, and suspension; these are assembled from smaller parts like pistons and the crankshaft, and so on. That's why we call this a "Software Factory": each node in the schema is not a single build or test, but a processing station that gets some materials and parts as input, takes some time to process them, and then outputs another part, which becomes the input for another processing station, and so on - until the ready-to-deliver product is produced.

Scalability and modularity

This high-level approach allows more focus on new features instead of maintenance: maintenance stays hidden within a “processing station”. Moreover, as soon as we start to think in terms of “processing stations” and “assembly line", we are automatically embraced to build modular and reusable components, consistent with each other – like gears in a gearbox.

In Software Factory, adding a component to your project is as easy as adding a workbench to a real fabric (and maybe even easier). Minimum necessary steps are:

1. Create documentation (requirements, architecture, test cases) ()
2. Implement MVP of the component – Can be automated using AI after completion of Step 1 ()
3. Add component description (repository, stages, environment etc) to project configuration file (manually; takes a few minutes including open text editor)
4. Run the Generator to update all the necessary configurations ()
5. Keep developing the project with the new component!

Benefits

Moreover, this approach encourages building modular components, which increases code reuse and decreases time-to-market.

Main benefits are:

• The same pipeline may vary depending on the components that are changed/affected.
• Each component may have its own set of quality gates
• Quality gates within one component are also flexible and may vary depending on the build type
• Interaction between components stays the same, as described in the project configuration
• Pipeline execution can and should be optimized through parallel builds and load monitoring & balancing

Speculative Merging

Because the pipelines are designed to detect defects as early as the commit stage-and then continue validation through nightly, weekly, and release stages-it is equally important to ensure that commits are merged in exactly the same order in which they were tested. Only that sequence represents a verified and valid system state.

If the tested commit chain differs from the merged commit chain, the validated state is effectively lost. For example, one subset of changes may be tested together, while another subset is merged in a different order or combination. In such cases, new interactions can be introduced that were never actually validated, creating the risk of cross-commit defects.

For this reason, preserving merge order is a critical part of maintaining pipeline integrity. It ensures that the state proven during Presubmit remains the same state that enters the main branch, preventing unverified change combinations from bypassing the intended quality gates.

The orchestration layer resolves this problem by maintaining a “green chain” of changes that are validated against one another. When a new change is ready for review, it is not tested on top of the current master state, but on top of the already validated green chain. This ensures that each change is verified in the exact context in which it is expected to be merged. The merge order then follows the chain from the bottom upward. In practice, once the lowest change in the chain has been approved by both Jenkins and a human reviewer, it is merged automatically. As a result, developers do not need to manage merge sequencing manually-their primary responsibility is simply to complete reviews promptly.

Beyond preserving technical validity, this flow also ensures that changes are merged in the same sequence in which they were developed and validated. This encourages teams to think in terms of the evolving system as a whole, rather than focusing only on isolated individual contributions.

One real-life example of a green chain can be seen below.

Load monitoring & balancing

The orchestration layer, together with Jenkins, already provides intelligent load monitoring and balancing to ensure efficient pipeline execution across available infrastructure. Instead of treating all jobs equally, the system continuously optimizes scheduling decisions based on execution history, platform characteristics, and current resource availability. This enables more predictable pipeline runtimes, better infrastructure utilization, and reduced waiting time for critical jobs.

This capability can also be extended to support hybrid execution strategies across on-premises and cloud environments. Based on current capacity, job priority, and predicted resource demand, workloads could be distributed across both infrastructure domains in a controlled and transparent way. In that form, the system would combine dependency-aware orchestration with infrastructure-aware scheduling, further improving scalability and execution efficiency as project complexity grows.

Watchdog & self-healing mechanisms

In a complex CI/CD environment, reliability depends not only on successful job execution, but also on the system’s ability to detect failures and recover from them automatically. For that reason, watchdog and self-healing mechanisms are essential for maintaining pipeline stability, reducing manual intervention, and preventing infrastructure issues from interrupting the development flow. To address this, the Reaction Engine was introduced - a service that continuously monitors system metrics and logs and reacts proactively to detected conditions. Depending on the situation, it can either notify the orchestration layer to recalculate load-balancing decisions or perform system-level corrective actions such as disk cleanup, service restarts, and similar recovery procedures.

AI Assistance

AI is a natural accelerator for the Software Factory approach because the entire engineering system is already described in text, structure, and relationships. In this model, AI does not act as an isolated coding tool. It becomes an additional processing capability inside the factory: one that can read project intent, understand dependencies, generate engineering artifacts, analyze results, and help optimize the overall flow.

This is possible because the Software Factory treats components as complete engineering units and manages the project through structured configuration, traceability, and automation. When requirements, architecture, tests, pipeline configuration, and results are available in versioned text form, AI can operate on the same source of truth as developers and CI/CD automation. For the underlying foundation that makes this possible, see the separate [ link to Everything-as-Code / X-as-Code section].

Key takeaways

Software Factory turns CI/CD into an intelligent orchestration layer for complex, multi-component projects. By understanding dependencies between components, it enables early defect detection, targeted validation, faster execution, and more efficient use of infrastructure.

Main benefits include:

• Better scalability for large projects
• Earlier defect detection through shift-left validation
• Faster pipelines by testing only changed components
• Clear responsibility split between project teams and Platform Engineering
• Flexible quality gates per component and build type
• Stronger traceability across the engineering lifecycle
• always green and release-ready code bas
• Better infrastructure utilization through parallelization and load balancing
• In summary, this approach improves quality, reduces overhead, and makes complex software delivery more predictable and scalable.