Maybe not microservice: The Case for Pipes, Pipelines, and Functional Isolation

1. Subsystem Decomposition

1.1 The Decomposition Problem

A subsystem decomposes a codebase into smaller, cohesive units. Two primary axes of decomposition exist:

• Technical axis: grouping by component type (controller, service, model, view)
• Functional axis: grouping by business capability (cataloguing, circulation, etc.)

1.2 Tension Between Framework Prescriptions and Decomposition Strategy

Organizing top-level subsystems functionally may create friction with frameworks that prescribe a technical-first structure. Concrete examples:

• Rails enforces model, view, and controller directories at the root level, making functional decomposition awkward without additional mechanisms like Rails Engines
• Sinatra (a microframework) imposes minimal structure, leaving architectural decisions entirely to the team

Frameworks with rigid prescriptions constrain architectural choices. Frameworks with no structure shift the entire burden onto the team with no guidance. This second approach might be fine for teams that know what they are doing and how to shape the architecture properly. Not everyone needs guidance from the framework.

1.3 Contexts as a Middle Ground

Phoenix provides contexts as a compromise:

• Explicit, guideline-oriented subsystems that enable functional decomposition without rigid enforcement
• Contexts define functional boundaries while allowing technical organization to remain nested within them
• Functional blocks may later evolve into microservices, but this is optional
• The same decomposition serves equally well in a modular monolith or a distributed architecture
• The choice depends on team needs, scaling requirements, and operational maturity, not on the decomposition strategy itself

2. Pipeline Topology and Data Flow

2.1 The Unix Pipeline Model

Unix pipelines model data flow through a single stream connecting stdout to stdin. This forms a linear chain where each stage's output becomes the next stage's input. Key characteristics:

• Each stage has exactly one input and one output
• Cognitive overhead is minimized because the topology is trivial to trace
• The linear, single-stream characteristic is not mandatory for a pipeline, but it reduces complexity significantly

2.2 Arbitrary DAG Topologies

Orchestrators like Airflow allow arbitrary DAG topologies with fan-in and fan-out edges. Tradeoffs:

• Powerful for expressing complex dependencies
• DAGs with dense interconnections tend to become hard to read even with visual rendering
• Complex function-call topologies with many parameters outside Airflow and Unix pipelines also produce unreadable code

2.3 Byte Streams and Opaque Containers

Unix-like pipelines connect programs by passing data through unidirectional byte-streams:

• Programs at each end agree on a structure such as JSON, CSV, or tar archives
• The pipe mechanism itself transports only raw bytes
• This has a direct analogue in dynamically typed languages: in Lisp and Clojure, collections (lists, maps, vectors) serve as opaque containers that can hold almost arbitrary data
• The consumer interprets the contents rather than the container dictating them

3. Typing Heterogeneous Pipeline Data

3.1 The Problem

Strict type systems introduce complications when handling heterogeneous data flowing through a pipeline where each stage transforms the shape slightly.

3.2 Failed Approaches

Single large type with many optional fields:

• Creates dependencies between all pipeline steps
• Loses the ability to reject illegal data
• Makes reuse difficult
• Changes to one field propagate everywhere

Many separate types for each step:

• Exhaustive and adds noise to the program
• Structures may not be mutually exclusive yet are treated as such
• Maintenance burden grows with every new stage

Both approaches fail because each pipeline stage depends on more than it needs.

3.3 Partial Fixes From Functional Programming

Two techniques alleviate but do not fully resolve the problem:

• Functional record update: enables creating modified copies without mutation, reducing coupling related to state changes
• Sum types: restore the ability to discriminate valid from invalid data and support exhaustiveness checking

Remaining limitation: every step that pattern-matches on a sum type must know about all variants. Adding a new case still propagates changes through the pipeline.

3.4 Structural Type Compatibility

Structural type compatibility offers a complementary solution:

• Independently defined types become compatible based on shape alone without requiring any inheritance relationship
• A consumer can specify only the subset of fields it needs via a structural interface
• Each step depends on a minimal projection of the data rather than the full type
• This decouples pipeline stages more effectively than either naive approach or sum types alone

3.5 Python Implementation

Python implements several of these patterns:

• dataclasses.replace(): supports immutable record updates
• The | union operator: simplifies union type expressions
• Protocol classes: enable structural subtyping, allowing independent types to satisfy contracts based on method and attribute signatures
• Tagged unions: modeled using Literal discriminator fields on dataclasses or TypedDicts
• typing.assert_never with mypy: enforces exhaustiveness checking on pattern matching or if chains, providing compile-time guarantees similar to sum types in functional languages

Combined approach:

• Protocols decouple steps through structural conformance
• Tagged unions enable variant discrimination with exhaustiveness checking
• Functional record updates reduce mutation-related coupling

4. Microservices, Processes, and Isolation Patterns

4.1 The Shared Principle: Isolated State by Default

A microservice and a Unix process share architectural similarities:

• Microservices: in well-designed architectures, a service does not share variables or databases with other services. Communication happens through well-defined interfaces. This is a best practice, not a hard technical constraint.
• Unix processes: each process has its own virtual address space and does not share memory directly with other processes. Explicit sharing is possible through mechanisms such as shm_open (POSIX shared memory) or mmap.

4.2 Historical Lineage

Microservices on GNU/Linux are literally processes communicating via HTTP over TCP/IP. The historical chain:

• TCP/IP: first implemented in 4.2BSD Unix in 1983
• HTTP: developed at CERN in 1989 to 1990, building upon these networking foundations
• Tim Berners-Lee wrote the first HTTP server and web browser on a NeXT workstation running NeXTSTEP in fall 1990
• NeXTSTEP was heavily influenced by BSD Unix
• GNU/Linux copied many initial ideas from Unix while remaining free and open

The modern distributed system traces an unbroken lineage back to Unix.

4.3 Pipes as an Alternative to Microservices

Piping via stdin-stdout chains is another mode of interprocess communication:

• Not as powerful or generic as TCP/IP or HTTP
• Easy to use and reason about
• Naturally fits data pipelines
• A data pipeline can be built using command-line tools piped together, running as processes on GNU/Linux instead of using microservices and a full orchestration system
• Scalability can be achieved by SSH and distribution through GNU Parallel, which launches jobs across multiple machines accessed over the network

4.4 Erlang and Clojure as Additional Isolation Models

Erlang processes:

• Lightweight alternative to Unix processes
• Rich high-level interprocess communication via mailbox message passing
• The Erlang VM enforces process isolation as a runtime guarantee

Clojure and other functional runtimes:

• Do not have the same process isolation constraints as the Erlang VM
• Provide lightweight isolation via persistent data structures and Software Transactional Memory (STM)
• STM allows memory sharing while preventing conflicts even when multiple functions run in parallel or concurrently

Bottom Line: Think Twice Before Going Micro

Here is the real talk. You might want to pause before spinning up your first microservice. Ask yourself these questions:

• Do I actually need physical isolation, or will logical separation suffice?
• Can a simple pipe between processes do the job just as well?
• Am I solving a scaling problem that does not exist yet?
• Do I have the ops maturity to handle distributed tracing, service meshes, and deployment pipelines?
• Will my team understand this architecture six months from now?

The truth is, Unix pipes have been doing data transformation reliably since 1973. Erlang processes have handled millions of concurrent connections since the 1980s. Functional isolation with STM has been working since Clojure showed up in 2009. None of these require Kubernetes. None of them need a dedicated platform team. And none of them will haunt you with debugging nightmares at 3am.

Microservices are not evil. They are just heavy. They are the nuclear option for isolation. Use them when the problem demands the weight. Otherwise, reach for the lighter tool. A pipe, a context boundary, a protocol type. Try the easy solution first. If it breaks, then scale up. Most teams never get to that point. And their systems stay simpler, cheaper, and easier to maintain because of it.

So yeah, think twice. Maybe thrice. Then build the smallest thing that could possibly work.