Backend Systems Architecture

What is Backend Systems Architecture?

The process of creating, defining and designing the architecture of a software system that is responsible for handling the data processing and logic of an application is known as Backend Systems Architecture.

When building a house, you would need a blueprint, if its a simple man-cave, maybe a simple drawing on a paper. Just to have measurments like how long the walls should be, how high the ceiling should be, where the doors and windows should be placed. Otherwise, you would end up with a mess. You would forget the plan in your mind after few days, maybe weeks. This draft you also save you from wasting time on purchasing items. You would know how many ft of lumber or number of bricks you would need.You can't just go to store and tell people "I need some wood and bricks to build a small mancave.". You would need to have at least a basic plan.

When build a software system, you would need a blueprint as well, and this blueprint is called architecture.

This blueprint is the design of the system, the structure of the system, the components of the system, the relationships between the components, and the behavior of the system. Altought requirements might change (will change), but its still crucial to have some sort of a well thought plan.

Why is Backend Systems Architecture Important?

When working with multiple teams, its important to have a common understanding of the system and needs. Every team have their own way of solving problems, and they might have their own way of designing the system.

Sometimes, the way different teams solve problems can be competible, but we can't be sure of that before we bring everyone to the table. For example, if the whole system runs on REST API, and one team decides to use another tech stack, like GraphQL, we need to ask, is it really necessary?

What benefits do we get if we let the other team use GraphQL? What are the drawbacks? Will this decision affect the other teams, create problems in the future? Make this system harder to maintain, find people to work on?

There are a lot of question to be asked. What if all the other teams OK with REST API, but are they OK with implementing parts of their system to GraphQL just because one team wants to use that?

When making decisions, we need to think about the whole system, and the people working on it. We need to think about the future, and the maintainability of the system. Because we can't rewrite it from scratch every time we need to make a change.

Creating up-to date documents, diagrams, and guidelines can help us to make better decisions and assure longivity of the system. Deailing with these steps can be time consuming and costing too much developer time, but when something goes wrong, or if people leave and new people join, having these documents will save a lot of time, money and maybe even prevent a rewrite or two.

For Many companies, the backend architecture might seem like an overhead, but think of it like insurance. Until you need it, every dolar you spend on insurance is a waste of money. But one day, you will need it, and you will be glad that you have it.

An architecture can write few pages of documents, and end up saving hundreds of developer hours without breaking a sweat.

My personal take architecture is the proper software manager. It makes sure that the system is built correctly, and it makes sure that the system is built in a way that it can be maintained and extended in the future. My honest opinion, I still don't understand many companies have more managers than architects. Some companies dont have any architects at all but have a lot of managers. To me, if a manager can do a thing that is important for the system, an Architect can do that and above. at very least, and architect will have a better understanding of the system than a manager.

Key princliples of Backend Systems Architecture

• Modularity
• Scalability
• Robustness
• Flexibility

Modularity

Each component of the system should have a spefict purpose. You need to ask and answer these questions. What does this component do, and why does it do it. If you can't answer either of these questions, you need to rethink the component. Its important to name these components correctly, and they tell what they do just by looking at their names.Sure, its fun to call them funny names, maybe dessert names or something (I'm looking at you Android), but it will only make onboarding and maintaining the system harder.

Scalability

The system should be designed to handle growth of traffic. If you have no customers, I don't think it makes sense to build a system for 10 million active users. Once you go from 0 to 10 million user, your whole system will be rewritten anyways. But that also doesn't mean design a system for 10 uses. Maybe try a large number like 3000-5000. Maybe thats all the customers you will ever have, who knows? But if you have a plan to grow, you need to design the system to handle that growth. Just be realistic. I have seen people design systems for up to million active users. When I ask total number of paying customers they have, the answer is usually less than 5. Okay. Maybe you are wasting your time and resouces for a problemn that is too far away.

Scalability is also thinking of when a task triggers, what else runs with it as well. For example, if you have scapper or some sort that gathers data from some site. That scrapper most likely trigger multiple events, like something needs to serialize that data, something needs to store that data, something needs to send that data to the user, maybe host the data as an API. But now, we need to understand what our bottleneck is. If our serializing is slow, it doesn't matter how fast we can store or send emails to the user.

If your cook is slow, hiring more waiters isn't the answer to your problem.

Robustness

The system should be able to handle errors and unexpected problems. Error handling is important. If you don't handle your errors, the errros will handle you. Its not uncommon to see a system with errors that no one knows about, because they all throw the same code. There are 100 parts of the system that can throw the same error, but no one knows where it comes from. So, if you have a system like that, which throws the same error for 100 different problems, do you really need that error handling, because it isn't handling the way you think its handling it. Its telling you "I'm fine", where it isn't, and it isn't telling you the problem either. If your system throws errors like "unknown error", that isn't error handling.

Flexibility

This is the hard part. The system should be able to adapt to changes. So when requirements change or time pass, adding or removing features won't break it compltely and not require a rewrite. Its easier to say than done and there are times it just won't work, because the how fondation was created, or maybe issues in the past weren't handled properly and teams just kicked the can down to road until the road ended. But its still important to think of this aspect. Rewrites are not uncommon and will happen regardless of how well you design the system, but it can be minimized and delayed with a good plan.

Common Challenges

• Complexity
• Adaptability
• Security
• Technology
• Resources
• Stakeholders

Complexity

The system should be as simple as possible. It's easy to design a complex system because if you'd thought it through enough, your solution wouldn't be a spaghetti mess to begin with.

So, it's actually hard to design simple systems because you need to understand the whole picture and even visualize it. We all can see a certain part of the bigger picture. The more of it you can see, the simpler you can design it because more parts will click together when you look at it.

Some people tend to design systems as complex as possible and feel good about being able to do it that way. But that shouldn't be the goal. If you can take a design, reduce its pieces, redesign what's left, and repeat this cycle a few times, you'll end up with a better solution than what you started with.

It's a different approach. The common approach is to build something, make it more complex, and keep repeating that cycle. Ask yourself: which way would you rather work and support?

The more you distribute, the more often you spread out, the more complex the system becomes. It's like a bunch of wires going everywhere. The longer the wires are, the more likely they are to get tangled. And sometimes, to untangle them, you either need to cut a few to free things up or disconnect them all and start over.

If I design something and present it for 10–15 minutes to a group of people, and no one understands a thing, who should we blame? Maybe Steve Jobs for the iPhone, since people's attention spans have gotten so short that anything over 2 minutes feels like too much. Or maybe, just maybe, the design is too complex.

You also need to consider the developer power and the budget you have. A 3-person team with a $100-a-month budget running on AWS Lambda and utilizing Kubernetes? Yeah, bud, wake up.

Adaptability

The system should be able to adapt to changes. Tightly coupled systems are hard to change and each new feature or change can break the whole system.Thats a no bueno. Sometimes, team take the short cut and say "lets just implement this and we will fix it later". I haven't seen this "later" yet. Some other teams say "Lets just ship this, and fix it in phase 2". There is a magical place, called the Phase 2 island, where all the bugs and issues are fixed. But no one has ever been there. There are people who claim they have been there, but they are usually the same people who claim they have met Santa.

Some devs like to try new tech—they enjoy the challenge. But is the tech they want to use really the right answer? Maybe. How long is it going to take, and who’s going to maintain it? Why can't we use the old Tech that everyone is familiar with?

There are 5 fundamental questions in life and software development:

1. Who started it? (Who wants this?)
2. Are we going to make it? (Skills, budget, time)
3. Where are we going to put it? (How will we use it?)
4. Who is going to clean up? (Who will maintain it?)
5. Is it really needed?

You need to answer these questions before doing anything. Imagine you can answer the first four questions, but your answer to the fifth is “no.” Why bother?

Security

Simply put, security is important. Why? Because if you mess this step up, goverment and lawsuits can mess you up in return. We typically don't like that. There are usually legal requirements for security, how you store data, how you handle data, how you transfer data, and several more how's. It depends on the county you are doing business in, and the country you are storing data in.

EU tends to over regulate but fines are low. US tends to under regulate but fines are high. Asia and middle east another story. So you need to be aware of the legal requirements and make sure you are compliant. But if you system doesnt have millions of users, you typically will use 3rd party services, like auth with Google, or store data with AWS.

These services usually handle the legal part for you, but you need to make sure you are using them correctly. If you store important data in a public bucket, problems will arise. Common sense and bit of reading the basics, maybe doing some Google searchs and use is the solution you might be looking for.

Technology

First step is to know what you team knows and what they are comfortable with. The cutting edge new techs are always good, but are there anyone in your team that knows that cutting edge.

Curse of the sotware development is there are millions of different ways to accomplish the same thing. Some ways are better than others, but it doeesn't matter if your team doesn't know those better ways. At the end of the day, you need to ship. If going with the bad ways to solve the particular issue is the only possible way to ship, then you know which way you need to go.

Resources

Resources can mean anything, engineering time, manager planning, budget, hardware, software, etc. List is endless, because depending on the project, resources can be anything, number of birds that land on your window can be a resource, if you are building some sort of a image recognition system.

Usually, the upper management people, who don't understand software say, "we can do this we have the engineers". Sure, you can. But do you have the amount of engineering time required to finish it on time?

Think about it, its simple to build the front page of the amazon site. I can do it, maybe in a day or two? Can I dream to build the back side of the amazon site in a day or two? Not even my dreams. That requires a team, maybe millions of hours of engineering time. So, you need to be realistic about the resources you have and the resources you need.

Everything looks simple when you aren't doing, I see people yelling at soccer players for missing a goal, if I was there I would have scored that. I don't know, I have my doubts.

Last month, one of the upper managers suggested we build our own local AI helper. I said, sure, we can—but we don't have the money for it. People who aren’t technical often think that if you put five people in the same room, you can build anything. Well, it depends. Are those people paid fairly in their own eyes? Is the project interesting to them? Do they have any family-related issues at the moment, like sick parents or newborns?

You need to be lucky to answer all of these questions positively. Everything might look perfect, but if half the developers feel they aren’t paid enough, they won’t be motivated to work on the project. You hired them to work on a specific task and agreed on a certain pay. When you come up with a completely different task and offer the same pay, it’s really hard to motivate them.

Why? Because everyone knows what that work is worth. Upper management often assumes, “They’re just pressing keys and looking at things—how hard can it be? Why do we need to motivate them for a new project?” News flash: people have feelings, and they have their own lives.

Stakeholders

Before making changes, the affected and interested stakeholders should be informed. If not, depending on the nature of the change and the company culture, you could find yourself in some very heated discussions.

I’ve always found it easier to have these conversations with technical people. You explain to them what’s going on, why the change is needed (or not needed), and they contribute their ideas. Sometimes you’re right, and sometimes they are. Which is fine—if it saves time and results in a better product, who cares who’s right?

With non-technical people, it’s a different story.

Here’s a true story from my work life: A non-technical person decided to move one of our most-used internal applications from C#/ASP.NET to React and TypeScript. The developer tasked with making this change is in his 60s, planning to retire in a year or two, and has no idea what TypeScript or React are. He’s an expert in ASP.NET—he’s been coding .NET longer than I’ve been alive, but he’s never seen TypeScript.

We tried explaining this to the non-technical stakeholders. I really tried. Their response? “He can learn, can’t he?” Well, sure, he can, but he won’t. He’s not interested in learning new technologies. He’s interested in retiring. We ended up wasting about a year on this project, and it’s still not finished at the time of writing.

I suggested hiring a React/TypeScript developer to at least get the ball rolling, but stakeholders and upper management said, “We have enough developers.” Maybe that’s true, but we didn’t have developers experienced in React and TypeScript. I was the only person in the company who could do it, thanks to my personal interest and involvement in other projects. I offered to help, but then who was going to do my job?