BuyAllMemes Blog

"I knew you'd say that" - Judge Dredd

After publishing Practical Dependency Inversion Principle article, I received amazing feedback from one of my dear colleagues.

It was in the form of a question:

...there is another problem, the cross-dependency between modules/packages.

What are your thoughts on this?

The question was premised on the schema that looks like this:

With code structure like this:

├── test
│   ├── notification
│   │   ├── NotificationUser.java
│   │   ├── NotificationUserRetriever.java
│   │   └── NotificationModule.java
│   ├── user
│   │   ├── UserModule.java
│   │   ├── UserNotificationRetriever.java
│   │   └── UserNotification.java

Where NotificationModule implements UserNotificationRetriever and UserModule implements NotificationUserRetriever.

It's not that hard to imagine:

• NotificationModule wants to know something about a user, and the dependency on UserModule is inverted, exactly as it should be
• UserModule needs something from NotificationModule, and the dependency is also inverted

This is what's called Circular Dependency.

And it's extremely problematic. Dependency Inversion ultimately plays no role here, even with direct uninverted dependencies such a case can occur, and the Dependency Inversion Principle by itself cannot fix it. Some frameworks (like Spring) and build tools (like Maven) will produce an error in case even a single circular dependency is detected. The main reason is — it's just too dangerous to resolve. It's a recursion. Unless treated with care it can produce such nice things like out-of-memory, stackoverflow, etc.

But, more than anything, it reveals the fundamental flaw in the system design.

In this article, I'm going to share some tips-and-tricks on how to treat circular dependencies. And I'm going to start with the most radical one.

Tactical Merge

Yes, I know. You are your colleagues spent weeks and months trying to separate UserModule and NotificationModule. You might have even extracted them into systems separated by the network to enforce sacred domain boundaries. And now I'm suggesting to move everything back together into a single SpaghettiModule? Hell no!

Hear me out. The software is supposed to be... soft. Flexible. Like clay. The purpose of the software is to help businesses achieve their needs. If the software is designed in a way that does not allow developers to build certain features effectively - the design is a massive failure. At the end of the day, most product companies are not selling their software directly, but rather via a service that software implements a.k.a. SaaS. I think we can agree on that.

For example, do you care about the system design behind a google.com? If you're a nerd, maybe. A regular person cannot care less about the underlying software. But everyone cares about this software working. Everyone.

So yeah, if UserModule and NotificationModule want to be together, because business requirements want so, it's probably a good idea to consider merging them, and reshaping into a single domain. Don't feel overprotected by existing boundaries. Sometimes mistakes are made, and the worst thing we as engineers can do is to be stubborn about it.

It's a very humbling experience. You should try it.

One direction

A less radical, but a bit more political approach is to invert dependency only from one module to another, and leave the direct dependency from another module back.

For example, we decide that NotificationModule is the high-level module, and UserModule is... well, further from the core of the business logic. This is where the political card has to be played because the team that manages UserModule might not agree on doubling down on NotificationModule dependency:

With the code structure like this:

├── test
│   ├── notification
│   │   ├── NotificationUser.java
│   │   ├── NotificationUserRetriever.java
│   │   └── NotificationModule.java
│   ├── user
│   │   ├── UserModule.java

And so there we have it. UserModule directly depends on NotificationModule, and there's an inverted dependency from UserModule to NotificationModule. The dependency cycle no longer exists. At least, during build time. There's still the possibility of an infinite loop during a runtime:

• NotificationModule invokes a NotificationUserRetriever interface that's implemented within UserModule
• To implement NotificationUserRetriever UserModule needs something from NotificationModule and so it calls it directly

This is more like a hack or remedying the symptoms. The disease is still there. Modules are still tightly coupled. Domain boundaries are wrong. We just tricked the system.

To solve this problem once and for all, one of the dependencies has to be broken. The best-case scenario is that both of them no longer exist.

However, there are ways to break circular dependencies via some integration patterns. Queue is the first thing that comes to my mind. Is it possible to eliminate the dependencies altogether by listening to a message queue? Or maybe something a bit more robust, like a Kafka topic? Sounds great! Don't. It's even more dangerous.

Let's go through a "hypothetical" example:

• NotificationModule receives a request from out there, and after fulfilling the request, it emits an event to UserModule
• UserModule receives an event, performs some computation, updates some user data... and sends an event to NotificationModule
• But, unfortunately, when NotificationModule receives an event, and after performing some computation, it decides to notify UserModule via event

You can see where it's going. The system ends up in an asynchronous loop of events exchange that never terminates. It might go for days and weeks unnoticed. Until, eventually, with more and more requests triggering infinite loops, the whole system will grind to a halt and go OOM.

Been there. Done that.

Extract new module

This is a tricky one because it's very easy to get it wrong and make things worse.

The approach is to extract functionalities that produce circular dependencies into a new even more high-level module. And invert the dependency from it.

The code structure:

├── test
│   ├── aggregator
│   │   ├── AggregatorUser.java
│   │   ├── AggregatorUserRetriever.java
│   │   │  
│   │   ├── AggregatorNotification.java
│   │   ├── AggregatorNotificationRetriever.java
│   │   │
│   │   └── AggregatorModule.java
│   ├── notification
│   │   └── NotificationModule.java
│   ├── user
│   │   └── UserModule.java

We're demoting UserModule and NotificationModule to a lower level of abstraction, and introducing a new higher level AggregatorModule (naming is hard).

So that NotificationModule depends on AggregatorModule, and UserModule depends on AggregatorModule. The nuance here is that AggregatorModule now exposes two interfaces, but NotificationModule and UserModule can cover only one of those each, so the setup requires more attention.

There are whole lots of tricks that could be applied to handle such a case: from something like a combination of @ConditionOnMissingBean(...) and @Primary bean annotations if we're talking about Spring Framework, to something as simple as the default interface method. And if you feel like there might be more modules to depend on AggregatorModule it might be a good idea to introduce a generic aggregator interface. This is where the real engineering begins.

This approach seems like a quite straightforward one. What's easy to get wrong here? I'm glad you asked. And the answer is simple — direction of dependency inversion. It might sound like a brilliant idea to introduce AggregatorModule and to make it depend on both UserModule and NotificationModule:

With code structure like this:

├── test
│   ├── aggregator
│   │   └── AggregatorModule.java
│   ├── notification
│   │   ├── NotificationUser.java
│   │   ├── NotificationUserRetriever.java
│   │   └── NotificationModule.java
│   ├── user
│   │   ├── UserModule.java
│   │   ├── UserNotificationRetriever.java
│   │   └── UserNotification.java

AggregatorModule implements both interfaces. UserModule and NotificationModule no longer know about each other. Sounds great! Except it's not.

Where AggregatorModule will get the information to implement NotificationUserRetriever for example? From UserModule of course. And what about UserNotificationRetriever, how to implement it? Invoke NotificationModule.

So the more realistic dependency schema should look like this:

So instead of one circular dependency between UserModule and NotificationModule, there are two, and they are even more distributed! And, the best way to solve a problem is to distribute it. COVID? Anyone?

So yeah, be careful. In this case, inversion of dependency could do more harm than good.

And this is exactly why I started with the Tactical Merge. Although it seems like the most extreme, it guarantees to work. Circular dependency will be broken.

Dependency Inversion Principle (DIP) comes from the famous SOLID principles, coined by Uncle Bob back in the 90s.

• Single Responsibility Principle
• Open-Close Principle
• Liskov Substitution Principle
• Interface Segregation Principle
• Dependency Inversion Principle

Food for the thought: Why are these principles bundled into SOLID, and not spread individually?

Most developers have heard or read something about them to some extent. From my experience, most devs (myself included) stop after SO, leaving LID for later days, because they are confusing.

• Who the hell is Liskov? And whom she's substituting?
• Why do we need to segregate anything — isn't it a bad thing these days?
• And which dependencies should we invert and how? And what about dependency injection?

In this article, I'm going to shed some light on the Dependency Inversion Principle, since it's the most impactful and addicting, in my opinion. Once I've started inverting the dependencies in my systems, I can't imagine living without it anymore.

Naming things

There are only two hard things in Computer Science: cache invalidation and naming things.

So let's deconstruct the name: dependency inversion

Dependency

Having a dependency implies that we have at least two of something, and there's a dependency between these somethings. What is something? It could be anything really; the only restriction is that this something is somehow bound by its context. It might be a single class, a package, a component, a group of packages, a module, or even a standalone web service. For example, code that calls the database to fetch a user. There are many possible names for such a thing: domain, module, component, package, service, etc. Name is unimportant, as long as it's consistent throughout the discussion. I'll call it a module. A module that queries a user from somewhere (presumably DB) - the UserModule. That's the first. But we need one more. Let's say we want to send a user notification because an appointment with a doctor is confirmed. And here we have our second module — the NotificationModule.

The code might look something like this:

package test.notification;

import test.user.UserModule;
import test.user.User;

public class NotificationModule {

    private final UserModule userModule;

    public NotificationModule(UserModule userModule) {
        this.userModule = userModule;
    }

    public void sendNotification(long userId) {
        userModule.findUserById(userId)
                  .ifPresent(this::sendNotification);
    }

    private void sendNotification(User user) {
        // notification logic
    }
}

package test.user;

public class UserModule {

    public Optinal<User> findUserById(long id) {
        //fetching user from the DB
    }
}

package test.user;

public class User {
    private String name;
    private String surname;
    private String email;
    //50 more attributes, because why not
}

Folder structure:

├── test
│   ├── notification
│   │   └── NotificationModule.java
│   ├── user
│   │   ├── UserModule.java
│   │   └── User.java

According to the code NotificationModule depends on UserModule.

Such code could be found everywhere. I would go as far as to say that 99% of the code I've read(and written) looks like this. And it might seem that there's nothing wrong with it. In the end, it works, it is straightforward to read and easy to understand. But there's a problem. Our sacred logic of managing notifications is polluted with something we don't have control over. Notice, that UserModule resides in a different package than NotificationModule. It's not a part of the notification domain. It's a domain on its own.

From the perspective of the NotificationModule, the UserModule is a low-level implementation detail. And this detail is leaking more and more into the module that depends on it. See the User class? It's part of the UserModule, not the NotificationModule. And NotificationModule is just one of its clients. Obviously UserModule is used throughout the system. It's the most used module in the whole system. Everything depends on it!

But wait. Why would NotificationModule care about where the user is coming from? It just needs some of the user data, and that's it. The concept of the user is important, but not where it comes from. And what if a User object is large, but we need only a few fields from it? Should the new SmallUser object be introduced near the UserModule? Isn't this a circular dependency then? NotificationModule depends on UserModule in code, but UserModule depends on NotificationModule indirectly logically? It's not hard to imagine how this goes out of hand. I've seen it go out of hand. Every. Single. Time. I've seen with my own eyes systems being tied into knots by such modules. And months and months of refactoring spent just to be reverted with "It's too much. Too expensive. Not worth it." comments. I wrote such systems.

The root of the problem lies in the dependency direction. High-level NotificationModule depends on low-level UserModule. Level in this case means the level of abstraction. The further we go from the edge(domain boundary) of the system — the higher we go in terms of abstraction. For example, modules that talk to DB are on the edge of the system (the scary network), so as modules that send HTTP calls, talk to message brokers, etc. However, the modules that prepare notification messages are much further from the edge of the system, so the level of abstraction is higher. It's a relative term. Like Java is categorized as a high-level programming language, based on its proximity to the bare metal, in relation to something like Assembly language which is the lowest of them all.

And so the dependency tree might look something like this:

Dependency direction goes with the direction of an arrow. Everything directly or transitively depends on UserModule. The core of the system is not the business logic, but the module that retrieves a user from the DB. This is fundamentally wrong. We want the business logic to drive our system, not the I-know-how-to-talk-to-a-database-thingy.

Inversion

This is pretty much self-explanatory, or so it seems. Google tells me that inversion is a result of being inverted. Thank you, Google. And the verb invert means put upside down or in the opposite position, order, or arrangement. There it goes, putting upside down the dependency, so that it's no longer A->B, but A<-B. But how to achieve this? We don't want UserModule to call NotificationModule to send notifications about appointment bookings, it makes no sense. What we actually want to do, is to make UserModule depend on NotificationModule, but not interact with it.

How?

Are you watching closely?

Interfaces. Take your time and look through the refactored code:

package test.notification;

public class NotificationModule {

    private final NotificationUserRetriever userRetriever;

    public NotificationModule(NotificationUserRetriever userRetriever) {
        this.userRetriever = userRetriever;
    }

    public void sendNotification(long userId) {
        userRetriever.findUserById(userId)
                     .ifPresent(this::sendNotification);
    }

    private void sendNotification(NotificationUser user) {
        // notification logic
    }
}

package test.notification;

public interface NotificationUserRetriever {
    Optional<NotificationUser> findByUserId(long id);
}

package test.notification;

public record NotificationUser(String name, String surname, String email) {
}

package test.user;

import test.notification.NotificationUserRetriever;
import test.notification.NotificationUser;

public class UserModule implements NotificationUserRetriever {
    public Optinal<NotificationUser> findUserById(long id) {
        //fetching user from the DB
        //and maps it to NotificationUser
    }
}

Folder structure:

├── test
│   ├── notification
│   │   ├── NotificationUser.java
│   │   ├── NotificationUserRetriever.java
│   │   └── NotificationModule.java
│   ├── user
│   │   └──  UserModule.java

There is a huge fundamental difference. NotificationModule no longer depends on UserModule. There's not a single import statement from test.notification that points to the test.user package. Not a single one. NotificationModule knows nothing about the existence of UserModule. NotificationModule is decoupled from UserModule, but not the other way around. It just asks the universe(system) for a NotificationUser using its own declared interface NotificationUserRetriever. And the universe(UserModule) answers. This is its job. This is what this module does. It abstracts the database on behalf of other modules.

And so the direction of the dependency between NotificationModule and UserModule is inverted. Given that we apply the inversion to all dependencies; the dependency tree might look like this:

Not only does the system no longer directly depend on UserModule. But the transitive dependencies are also much more relaxed.

What if UserModule grows out of hand? We can re-implement some interfaces in another NewUserModule without affecting anything. There's no god User object to grow out of hand. Instead, there are several domain-specific representations of a user, which have no dependencies between each other whatsoever.

But every decision is not without tradeoffs. In the case of dependency inversion, the tradeoff is the amount of code. If every module that wants to retrieve a user introduces its user model and an interface to support it, UserModule will grow pretty quickly. And most of the code will just map a database object into yet another domain object. It's not the most exciting code to write or to test. UserModule is no longer threatened as the module, which everyone has to bow to and respect, but rather the mere mortal boring worker. And it works. But as I've mentioned before, nothing stops the refactoring of UserModule into several smaller more exciting modules, each implementing its interface and fetching only what's necessary from the DB. And some of them might talk to something else, like a cache, another service, go for another DB table, etc.

One more thing

The Dependency Inversion Principle scales far beyond a couple of simple modules. It's extremely powerful and addicting. But it's important to know where to stop. Some literature states that everything should be abstracted and inverted. Including frameworks. I think this is an overkill. Abstracting the DB engine and inverting the dependency on it is a good idea. Running around abstracting the framework of your choice, because someone from the internet says so, is not the smartest idea. It's a waste of time. For example, Spring Framework (so as pretty much every web framework nowadays) provides amazing capabilities of DI (dependency injection, not inversion) that enable performing Dependency Inversion almost effortlessly. Almost.

It requires practice though. Quite a bit of practice. And it feels weird at first. Because we're so used to envisioning systems as three-tiered which goes from top to bottom or from left to right — A->B->C. In reality, systems are more like a graph, where dependencies are pointing inwards to the business logic — A->B<-C.

You guessed it right: Clean Architecture, Onion Architecture, Hexagonal Architecture and such are ALL based heavily on the Dependency Inversion Principle. These are different implementations of DIP. But before you step into one of those architectures and claim yourself an ambassador, I would suggest stepping back and practicing DIP on a smaller scale.

Refactoring

Last but not least. Dependency inversion is an amazing refactoring tool. And it doesn't get enough credit for it.

Let's imagine, the system is not a greenfield. Let's imagine, the system is 7+ years old. The UserModule from above now contains several dozens of public methods and has a dozen other dependencies. The User object contains about 50 fields. Half of them are weirdly named booleans. There are quite a few complex relationships.

And here we are, building a brand-new notification system. And we need some information about the user. About three-four fields.

We have two options, and two options only:

1. NotificationModule depends on UserModule. We reuse one of the existing public methods from UserModule to fetch a User object. Then we perform all the necessary transformations on a user within the NotificationModule, and that's it. The job's done.

   But we're added to the mess. UserModule now is a bit harder to refactor, because there's one more dependency on it. NotificationModule now also is not that new. It's referencing a huge User object left right and center. It's now the part of the ship. Maybe you would like to introduce yet another method to UserModule that returns a smaller user? And now there's even more mess.

   How do you think those several dozens of public methods were added? Exactly like that.

2. Inverse dependency. We are not going to allow mess into our new NotificationModule by any means necessary. Our new module is too innocent to witness the monstrosity UserModule has become. Instead of depending on a mess, we're going to inverse the dependency and make the mess depend on our new slick domain-specific interface. The mess is still there, but we're not adding to it, which by definition means that we're reducing it. At least, within our new NotificationModule. And when someone eventually decides to refactor UserModule, all they need to do is keep the interface implemented. Not the several dozens of public methods with unknown origins introduced within the last 7+ years. But a single interface that leaves within NotificationModule domain.

I don't know about you, but for me reducing the mess beats adding to the mess any day.

So, the tech. Oh yes, the most important part — the tech. I'm going to use stuff I'm most comfortable with, which is happened to be the most widespread tech stack in the world: Angular frontend, Java + String backend, and all that on top of AWS.

Let's begin with infrastructure — to keep things simple, I'm using AWS Amplify to run frontend, and AWS AppRunner to run backend. For now, there's no need for anything more complex than this.

AWS Amplify

AWS Amplify hooks up to frontend repository via GitHub webhook. And everytime anything is pushed into main branch, Amplify gets notified and the CI/CD machinery kicks in. Amplify is smart enough to understand that it's connected to angular app (this actually doesn't matter, because it builds a project with a silly npm run build script).

Build artifact is then stored into AWS S3 bucket (unfortunately, or not, this bucket is not accessible) and then exposed via cloudfront distribution(also not accessible). By "not accessible" I mean that it's not created under my account, I can't look at it nor touch it. It exists, but somewhere within the bowels of AWS. Serverless, right?

AWS S3 is a perfect place for frontend artifacts – infinitely scalable, ultimately robust, publicly accessible(when needed), cheap. It just works. I have a strong impression that AWS S3 powers at least half of the internet, and so I'm trusting it to host my amazing frontend.

AWS AppRunner

After the first blog post, I had no backend for my blog application.

— "Do I even need a backend?" - was my question.

— Of course, I'm a backend developer, I have to have a backend.

— Alright, let's have it.

Building backend is straightforward. Code here, code there — I'm doing this for the last 15 years, so I'm feeling somewhat comfortable. The real question is "How to run it?"

EKS? Hell no, I'm not touching Kubernetes. I'm sick of it. It's too complex. Moreover, I want to run a single container. To say that EKS is an overkill in this situation is a huge understatement.

ECS? Sounds better. Let's do it. I've created a cluster, task definition, created a task... and nothing. I can't access my service from the outside. Oh, no... networking. Something is not right with the VPC setup. Subset seems fine. Security groups and routing tables are also "looks fine." Damn it, something silly is not right, and I can't find it. Screw it — a task stopped, task definition deleted, cluster deleted. ECS is also too complex.

While in bed and half asleep, I was browsing through AWS Console app on my phone. Eureka! AWS Q. AWS AI assistant. This is exactly what they built it for — so that idiots like me could ask questions like mine. The answer was instant — AWS AppRunner. Next morning I logged in into AWS AppRunner, clicked a few buttons, selected a hello world image from ECR, "deploy" and... it worked. My hello world backend is running in a matter of 2–3 minutes. This is why I love AWS.

I've hidden my app deployment via a custom domain api.buyallmemes.com by fiddling with Route 53 hosted zone. Thankfully, I know a couple of tricks around DNS.

Now, it's time to build the real backend.

Java + Spring = ❤️

The choice of tech for the backend is super easy. There's no choice really. There's only one true kind, and it's Java + Spring. I'm starting with an extremely simple setup: one REST endpoint that returns a list of posts. What is a post? A simple resource with only one attribute — content. For now, I don't need anything else.

However, I do need something — Zalando Problem library https://github.com/zalando/problem. I'm sure you're aware of Zalando as an internet cloth retailer, but you might not be aware that they have quite a few cool bits of software. Problem Library is one of those bits. It's a small library with a single purpose — unify an approach for expressing errors in REST API. Instead of figuring out every time what to return in case of error, or returning gibberish (like a full Spring Web stack stace in case of 500), zalando/problem library suggests returning their little Problem structure. Naturally, a library has an awesome integration with Spring, so there's very little configuration required. Use it, do yourself (and your REST API consumers) a favor.

Another one of those hidden gems is a Zalando RESTful API Guidelines https://opensource.zalando.com/restful-api-guidelines/ — read it. It's awesome.

So, after the initial setup, I throw a bunch of code in.

Rule #1: First, make it work, then make it right, then make it fast.

I don't care about performance at the moment(if ever), so I will ignore the latter part. Let's focus on making things work.

Damn it, I need a database to store posts! Or do I? Hmm, why the hell would I need an enterprise grade DB (like PostgreSQL) to store a single post - sounds absurd. I will store it on disk as part of the source code! My IDE is a perfect .MD editor. Git will provide me with all the version control I ever need. I can just branch out of the main, write whatever I want, and then merge it back when it's ready to be published. And it's free!

Well, I need to redeploy the backend every time I write or change the post, but for now, this is not a big deal, so this mechanism will suffice. I've set AWS AppRunner to automatically detect and deploy the newest image versions of my backend. So I don't have to do much manual stuff, besides building an image.

Btw, how do I suppose to build and push image into ECR? I'm not writing Dockerfile — that's for sure. Google Jib, https://github.com/GoogleContainerTools/jib.

Simple jib gradle plugin declaration in build.gradle(Gradle FTW!), set jib.from.image parameter to amazoncorretto:21-alpine, set jib.to.image to my ECR repo. Quick aws ecr get-login-password... from ECR documentation, ./gradlew jib and off flies my images. Easy enough. I will automate it later. I think GitHub Actions is what cool kids are using (I'm more of a GitLab user, but for the sake of exercise, I decided to publish everything on GitHub).

Alright, for now, that's enough. I have a running Angular frontend and Java backend. Frontend knows how to talk with backend. Backend return list of posts, which are store in resources folder. Backend logic is rather silly

• Read files from resources/blog/posts project folder
• Load each file content as string into a Post object
• Sort loaded posts by filename in descending order

And yes, I've introduced fileName attribute to the Post. And that's about it. I already established minimal flow of work.

At the moment, there's little to talk about. There's little code and one cute unit test. I guess this is worth talking about — I'm a huge fan of TDD. I love my tests. At the moment, I have only one but crucial test that covers two most important aspects — REST endpoint and that posts are properly ordered. I decided to use file naming as a sort parameter. Each new post-file will be prefixed by the current date, so I could easily sort them in reverse order to show the latest posts on top, and oldest at the bottom. Since I'm a backend guy, I prefer to keep such logic at the back. I don't want to spend much time on frontend, so I will try to keep it as lean as possible. Saying that, the more I think about it, the more I realize that I should've gone with something like a thymeleaf, and build everything within the backend app, but what's done is done. Having a separate frontend app is not without its benefits anyway. Plus, I can definitely benefit from expanding my horizons beyond backend and Java.

I hate frontend. But at least, I figured out how to use markdown to render content, so I don't have to struggle with WYSIWYG editors, at least now.

But where was I... Oh yes, BLOG! I'm building a blog - something you've never heard of or seen before, right? I hope you can read through my sarcasm, I'm using it a lot, and I'm not going to tell you where - figure it out by yourself.

The idea is straightforward — share my knowledge, thoughts and opinions on software stuff. And there's no better way to do it, but via examples. So, let's do it!

I'm going to build a blog while covering certain aspects of the building process in this blog. So you could see patterns in action. I'm going to start simple, heck, I'm a backend developer, who claims to be proficient in Java and distributed systems, but I'm writing this in .MD file, which I will copy-paste into a component file.

I want to make this process agile and iterative while doing only what is necessary to build what I want now. So, for now, it's a single-repo-almost-a-static-page-thingy - https://github.com/buyallmemes/blog.

Also, I kinda enjoy writing from time to time + I'm a programmer, so why not combine the best of both worlds — create a place where a can park some of my thoughts for good.