gears

Development Life Cycle. From happy idea to Production Safely

Posted on ddd, kubernetes, observability

I would like to cover from top to bottom the full development life cycle and technical stack I used in 2020.

Most of my career, I worked on web so this is heavily focus on that. Let’s start with the basics, the processes. I follow Domain Driven Design, which it’s end goal is to align domain and software and do so by aligning people. Let’s talk about the problem first.

Strategic design

Strategic design as mentioned in the Domain Driven Design book from Eric Evans, it’s a set of ideas and tooling focus on the business problem we want to solve. The place where this problem and the ramifications will live it’s called domain. In the book he presents a conversation about domain with the domain experts to discover the concepts behind that problem, group them into different contexts and define the relationships. It’s a conversation and maintain the same vocabulary for this concepts during the entire life cycle it’s critical. Communicate it’s critical in any business, and software development it’s not an exception. This conversations has a language to express the ideas but it’s evenly important the context where this conversations happen, as one thing can have a different name or a different meaning depending on the context we are talking. The language itself it’s that important that has his own name Ubiquitous Language, and it’s the language the business use and the people has to adopt. This language isn’t static and it will evolve as the conversations goes deeper and the people gets better understanding of this ideas, concepts and relationships.

As mentioned above the context it’s a key part. What a user can be for customer support can be totally different from what a user is for the chat. The first one needs way more detail than the second one for the support people to work properly and in the second one a user doesn’t even has to be registered in the site for example. How to name this concepts on each context will be naturally defined by having conversations and planning the projects. Define the limits of this context it’s a challenging task as some lines looks blur at the first conversions, but limits are required. This boundaries will define Bounded Context which, in an ideal case, should match with a subdomain and will contain the solution for this smaller problem. A subdomain has a problem, a bounded context holds the solution. You can see the ubiquitous language living inside bounded contexts and we can say that each bounded context has his own language. BTW, what’s a subdomain? Let’s explain.

At this point you may have an idea in your mind about what a subdomain may be but let’s explain anyway. Domain is where the business problem lives and subdomains are different areas of that problem where smaller problems lives. Example: You’re a gambling company, your problem is to make an online slots casino to make money, have fun, build a community, and your subdomains cloud be: customer support, game engine, bonuses, user accounts, forum, affiliates, payments… Etc. There’re 3 presented in the book

  • Core subdomains: What makes money, where you solve the problem. Where you want invest the most. (game engine, bonuses, user, payments)
  • Supporting subdomains: are the problem what support core subdomains but are not key. (forum, customer support)
  • Generic subdomains: Things you need but you don’t really care that much. (affiliates, legal regulations)

Topics already discussed

  • Domain and subdomains
  • Domain experts
  • Concepts
  • Ubiquitous language
  • Bounded Context

A business problem we want to solve will be explored by talking with the experts to extract the concepts and group them by context forming smaller problems in order to provide a solution for each of these.

A domain is discovered via ubiquitous language, subdomains star appearing while grouping concepts in different contexts that ended defining boundaries. Bounded contexts are defined to provide a solution for each smaller problem.

That’s my little resume of Strategic design. We can extract from this few conclusions about how to structure people:

  • Product people are key and his communication skills makes de difference.
  • Product owners should participate in the development lifecycle and not only planning and dailies, specially during the domain discovery phase and early stadiums of DDD implementation.
  • The closest to the devs as they can doesn’t mean rushing or micro management. Means open conversations, make sure concepts are properly assumed and correct when the language or the concept it’s wrong.
  • Subdomains can be challenging and complex as the problem they solve. Some can have multiple bounded context inside. Think in an e-commerce, shipping bounded context should know about some areas of inventory subdomain. There’re some parts of the inventory subdomain that will live in shipping bounded context.
  • Making people expert in different subdomains rather than projects/microservices will bring deeper understanding to the teams. This will impact in delivery as people already know about the problem and will iterate faster. Make teams own Subdomains not applications.
  • Structure your team however you want, choose your methodology but always keep in mind that the purpose it’s deliver as fast as you can without compromise reliability. It’s OK to push and deliver MVPs with certain quality measures, up to you to take the risk, but in the long run, iterate to remove technical debt must be a compromise.
  • Core subdomains is where you should put your best people and invest the most.
  • Generic subdomains are things to buy, avoid having this things in house as much as you can. Remember, maintain less non core components will provide more time for the heart of your business.
  • The goal of this it’s iterate faster in the medium – long run. A change in the paradigm requires time to be assumed, a common vision and evangelists that plan seeds across the teams. But, keep in mind that things that slow down and negatively impact in software development are: ambiguity and uncertainty. Which are the things we’re trying to solve before start. Deep dive in this great and detailed article.
  • When as developer you’ve a problem with naming, consult your product owner, not you peer. It’s a domain knowledge smell and may indicate a lack of understanding of certain concepts.
  • Devs will have tools to structure their code no matter the language, tests with enough quality and clear business understanding. That’s the first goal on the technical side, on the strategic side, devs should understand business priorities and do as much as they can to deliver faster on each iteration while product provides room for that in a win – win agreement. Of course this progression its asymptotic to a fixed number in the x axis.

And now that we’ve mention this we promise to devs, it’s time to talk about Tactical design, where things gets harder.

Tactical design

Tactical design is the solution space in software form. It’s composed of a set of technical resources with the objective of build the solution for a problem we encapsulated in a bounded context with the best clarity and expressed as in the ubiquitous language.

Above mentioned resources are:

Layer architecture

We’re going to split our code by responsibility depending how far is if from the domain. We’re going to use 4 layers (most people use 3 with User Interface as part of infrastructure, build your own criteria, both works OK). Also known as clean architecture and ports and adapters.

Domain layer

Where your business logic lives and the one that doesn’t interact with Any other layer, domain only cares about translate the language and the expected behavior into code. Implementations like persistence, open api spec, etc… are not responsibility of this layer.

Infrastructure layer

Where the technical implementations of the domain lives. Product says email should be stored and should be unique. Email value object lives in domain layer, EmailRepository it’s an interface that lives in domain layer, MySQLEmailRepositiry lives in infrastructure and implements the domain interface. We’ll use dependency injection to manage this in a clean way in the application layer.

Application layer

It defines every single use case you need to expose from you domain. It’s also where the infrastructure layer and domain merge composing the required technical solution for the problem defined in domain.

User Interface layer

This is how we expose or present to others. Can be a website, a message broker, an OpenApi http server, a console command. Everything that requires an input from the outside it’s considered User Interface.

Value object

I’ve mention before Email value object. I’m referring to a concept that only holds a value and the change in the value implies a new value object as this ones doesn’t have identity. If my personal email changes it’s because was replaced by a new one.

Entity

It’s any concept that has identity and changes in their properties doesn’t invalidate the entity. My car has a registration number, that can change if I sell the car, but also has a chassis number, which it’s unique to this car and any other has. I can sell it, repaint it, change the registration number, change the interior, the engine, the lights… But will still being the same car because it’s the identity what defines an entity.

Aggregates

Aggregates are conglomerations of highly coupled entities and value objects that represents a business unit. This aggregates, of course, are built by entities related to each other, where relationships are downstream or upstream, in which one of them will live at top and because of that assigned as Aggregate Root.

Aggregate Root

Will be the entry point for the rest of the entities that lives inside. Drive by example: a car. Chassis number makes the car identifiable, it’s an entity. Wheels, seats, bumpers, color, this are all value objects. The engine car engine number is different from the chassis one and it’s unique, so the engine will be an entity composed by tons of value objects that relate to each other. The a mechanic change the oil of your car, it’s an operation for the engine but for the car at the end. Car.engine.replaceOil() vs Car.replaceEngineOil(). The second one keeps private something that it’s an implementation detail or something you as owner outside the mechanical world, doesn’t care about. So in your context as owner second one makes more sense. In the mechanical would like F1, engines that important that can be the aggregate root of that conglomeration. Contexts.

Domain Services

When some domain logic depends on more than 1 aggregate it needs to live somewhere, this are Domain Services. This lives in domain layer.

Use cases

In clean architecture application services. Are core concepts in DDD and it’s the way we expose domain operations to the outside. Doesn’t contain business logic and generally are low complex classes. Are composed usually by Command or Query and the Handler, SignUpCommand and SignUpHandler for example. It’s also the place where we inject the infrastructure objects to merge logic and implementation.

Busses

Command and Query bus route each Command or Query to his Handler, usually passing through middlewares . It’s helps decoupling and testing but also provides tons of possibilities in the middlewares like ACL or caching mechanics.

Domain Events

Domain events capture the changes applied in the state of your aggregates. It’s also the way you’ll communicate between Bounded Contexts and are generally sent to an Event Bus right after state it’s successfully persisted.

You can find an example of how all of this concepts are implemented in this repository https://github.com/jorge07/billing-api.

DDD isn’t new but IMO isn’t extended enough.

Systems

Kubernetes, Kubernetes, Kubernetes, Kubernetes, Kubernetes…

Done. That’s the summary of my last 4-5 years. But let’s dive deeper.

Kops and Rancher

Best tools so far to create production ready clusters. Honorable mention for kube-aws which I used for years but the difficulties acquiring velocity with K8s upgrades and some breaking changes made some upgrades very difficult.

Considerations when Kuberneting

  • An upgrade plan it’s mandatory
  • Forget about kustomize, helm is the king and the facto standard. Things like rollbacks needs to be done asap and revert history and recreate previous the previous deployment can be very slow sometimes. Helm rollbacks are way better and already built.
  • Prometheus Operator. Yes or yes since day 1.
  • NodeLocal CacheDNS. Yes or yes.
  • Kiam/kube2iam. Yes or yes.
  • alias k=kubectl
  • One cluster, one kubeconfig. Prevent from switching to another terminal, switch cluster, back to the previous one and apply a change in the wrong cluster. Many times. I learnt, I share.
  • A SERVICE MESH ISN’T MANDATORY. UPGRADES BETWEEN MINORS IN ISTIO ARE A PAIN IN THE ASS. Semver, a beautiful lie.
  • Develop in minikube it’s so slow for most of my workflows. Docker compose it’s good enough.
  • Avoid Stateful apps in Kubernetes. Manage state it’s hard. Keep it outside to manage things like Kubernetes upgrade easier. Things like cost or legal can force you to use self hosted solutions instead SaaS, isolate them, avoid use it inside Kubernetes unless you know very well what you are doing.
  • Volumes, in clouds like AWS volumes are available per availability zone only and are immutable, you can’t move it. Once your volume it’s created, be sure your pod can only be scheduled in nodes on that availability zone or they’ll be in pending for a long, long time.
  • Do not require devs to know about Kubernetes, not too much. They don’t need it. Can be too extensive.
  • I still don’t know why deployment pod labels are immutable but deployment labels not. Are selectors, change the name
  • Since 1.18, you can list Ingress Class objects
  • Since 1.17 services can route the traffic by topology keys
  • Endpoint Slices can drastically reduce network traffic in medium-large clusters, make sure you’ve it enabled
  • Throttling it’s a big problem in kubernetes you must see this talk https://www.youtube.com/watch?v=UE7QX98-kO0
  • Readiness != Liveness
  • Memory Limit === Memory Request
  • Kubernetes events are very useful make sure you can query them when needed.
  • There’re too may CRDs. The less you’ve the less you’ve to upgrade in the future.

On calls

Make each time responsible of his owned context. Pay the on calls interventions an availability apart. Isn’t the same. Pay only for availability means infinite hours are covered. Pay only for intervention mean people availability is included in salary at anytime, and really, is not.

Remote

We’re going to be remote for a long time. Embrace video calls, talk about non work things too. Make a coffee call time to time with all the people that wants to join from any team. Make 1-1s a bit more often. Make trainings something mandatory every couple of weeks, we’re all too busy for many things but we like learn, people stay because they grow, it’s a win-win.

Non remote culture. companies will suffer from the need of control what people is doing, like if they were able to do it in the office. The transition to trust driven company will be very difficult for this kind of companies.

WIP