Diego Pacheco Tech blog

The Stability Mindset I have background on software development. Currently, I'm leading a team of highly effective engineering's where we are responsible of some of the components we delivery in production and I need to say that is not the standard reality on it market. Development is used to break things and move faster, however, lots of developers never cared about production or about “polishing” they systems. This effect happens some several reasons like: Having different people to develop and operate software Having different departments for development and Operations Having development teams focus too much in new features rather than real UX So there is this culture clash when we have developers doing DevOps Engineering(I say that for my own experience). We developers are not used to keep something running no matter what. To be always available and reliable. It takes some time to get this “Stability Mindset” but once you get it is hard to get out of you...

Last year I started to blog about something I'm calling "Stability Mindset". I'm still working on the ideas but I think I had enough time to time and experiences to evolve the concept to some basic guiding principles. You might be wondering if I'm just trying to do some kind of ugly re-branding of Reliability like Software Reliability Engineering(SRE) but I don't think is the case - I see is different things or at least a specific subset of SRE. If you are interested in a broader spectrum you can check out google insights on the matter. Google has this amazing book you can check it out about SRE. I thought a lot if this was just SRE for mere mortals but I don't think it is. I really think is a different kind of problem or at least different subset for sure. Today I'm writing this post because I want to capture some of my recent findings and also because I did a retrospective exercise with my team today I promise I you share some of my thoughts...

Kanban is a Lean tool. It's focused on the psychological part of the work. In a nutshell, you manage the system via constraints and you don't manage people. I used kanban a lot in the last 8 years. When I started I was a Software Architect | Coach in not-so-traditional SOA projects.  Today I work a lot with DevOps Engineering and Cloud Computing and I don't work with 2 weeks sprints but with quarters which are 3-months length buffers of work. You might think as a quarter as a super big sprint. There are several different kinds of classes of work like (A) Support for Developers, (B) Tuning, (C) Stress Testing, (D) NoSQL Development, (E) Telemetry and Observability, (F) Distributed Tracking and so on and on. I'm working on CORE team with other architect and DevOps Engineers. For a long time, I was using kanban metrics and sheets pretty religiously. Because I was working with Big Teams(up to 33 people) and multiple teams who had to deliver on time. As more close y...

Lessons Learned with DevOps Engineering Experiences Almost 20 years ago an Agile movement was born. After 20 years there are still lots of confusion and misunderstanding on principles, practices, and mindsets. DevOps barely have 10 years but does not have a method like a scrum, Kanban or XP. However, DevOps rely on Lean / Agile there is no strict definition, method or formal guidance how to do proper DevOps.   Like everything in life, this is good and bad at same time. DevOps is a movement about experiences that made some for some companies and people in order to have or practice better ways to develop and operate the software. Currently, DevOps is a mess. DevOps is a mess is Brazil but is also a Mess in the USA. For lots of people, DevOps means Ops doing code. For other companies DevOps means Cloud Ops Team doing ops. There are DevOps folks from Development background like me and there are much more. There is so much confusion on the market that you really need...

DevOps Engineering is a reality. DevOps principles like Immutable Infrastructure, Infrastructure as Code, GitOps and Observability are the default building principles for any cloud computing serious initiative. However, Infrastructure, provisioning, automation it's not trivial, tools like Ansible and Terraform make the engineer's life easier, productivity and fun. As time goes by we have more and more infrastructure code, how we make sure we keep software rolling faster with stability and quality?  The answer is testing.  Infrastructure testing is a very tricky, mocking terraform it's not easy. There are some interesting solutions like LocalStack ,  TestContainers , and molecule for Ansible . These tools are great but they are more towards Integration or E2E testing with are expensive, involve several other components and often super slow to run. A simple approach more towards unit testing could be achieved using Clarity .

Today I want to share some experiences I had in the last 2 years with AWS Lambda as a Serverless solution for Microservices Remediation. This was not a single effort from me only but a team effort. The lessons learned I want to share with you today was not only in sense of Serverless but also in sense of DevOps Engineering. DevOps Engineering is the norm today. I really can't see a world without it. Code it is the lingua franca of everything around IT. Currently, there are more and more abstractions and solutions rising for DevOps Engineering, chances are we will need code fewer things as the time pass. However, as we push the boundaries of innovation we face new problems and new problems will always require better solutions. DevOps Engineering is great however it's not a FREE LUNCH at all, like microservices, which are great too, there are lots of COST that are introduced. There is a requirement do structure and lay down teams in a different way and also there is a need ...

Observability is a must-have a property any mature distributed system solution and/or digital product.  There is no way to "buy" observability, you need to earn it. Observability is like car insurance. No one likes to pay insurance, however, if there is a care crash you do really really want it. However, you cannot acquire the insurance at the very moment of the crash it does not work that one. The car insurance metaphor would work partially If you have a monolith system. As you have microservices, therefore distributed systems, you have many more failure points and in order to detect that the "car crashed" could be much more hard and complicated. There are many challenges in order to have observability on the system. The main idea is that Observability is something LIVE which means it is not a one time job. So you want to have a better understanding of your system via instrumentation which will lead to better observability and this loop keep going. Some time abou...

Every company out there works with teams. There are all sorts of teams such as teams specialized in technology stack like Hadoop or teams which are domain-driven and have cross disciplines( Cross-Functional Team ) inside such as backend, frontend, QA, UX, Architecture, and more. People don't leave companies they leave teams, also it would be accurate to say that people don't join companies and they join teams. So should we have dynamic or static teams? Should teams self-assemble or should we have PMO doing that? Should the teams be stable or should we allow dynamic reteaming? Is work always fitting well in our team's structure? What about Platform vs product teams, which one is best? Should we just pick one model or should we use different models? Should engineers organize around managers or managers organized around engineers? Teams affect Architecture and our perception of ownership. Teams should be a people organization but often is just a grouping of people where everyb...

Architecture is about many things but I always believed * Design * was in the heart of architecture. However how we design something? There are many aspects to cover. In order to do better designs you need basically 3 things: * Problems to exercises your design skills. * A continuous learning process: Knows your tools, technology and always look for new approaches and techniques. * Review and Feedback process which could be done via a series of practices like Design Sessions, Design Review, working POCs.  It's also imperative to know your tools, what features do you have in your disposal, what techniques people often use, what principles can be applied and are often good ideas like Isolation, Testability, Debugability and which ones are smells or often anti-patterns like using Cassandra as a queue or Reflection in Java without caching . There are always macro and micro concerns you need to take into account. At the end of the day, small things can make a big difference in d...

Dyno-Queue is an interesting queue solution on top of Dynomite . Dyno-queue uses dynomite java driver a.k.a Dyno . Dyno Queue gets all benefits from dynomite like Strong Consistency, High Availability, Stability, High Throughput and Low latency and extend to queue semantics. I highly recommend you read Netflix post about dyno-queue.

DevOps Engineering is about Workflow Currently, I work doing a bunch of different things(like most of Brazilians). I do Architecture, DevOps Engineering, Software Engineering, Agile Coaching and a bit of management. There different types of companies and cultures but basically or you are more specialized or more cross-functional I think in general Brazil is similar like Europe and more cross-functional and the US is more specialized but really depends on the company. I also run a small team doing: Dynomite / DM , Serverless Remediation, Stress Test and Chaos Platform and Telemetry / Observability. My teams need to provide solutions(engineering)but also Stability .

  Building service is easy. Isolating that service, it's easier said than done. Anyone can build any service with any technology or stack. However, would this service last, or will it become technical debt? Many organizations trying to go all in with microservices end up with distributed monoliths instead of isolated services. The result is more complexity, less business value, and slower lead times. We had many movements and many different technologies and styles of architecture in our industry. From SOA, Soap, ESBs, BEPL, REST, Event-Driven, gRPC, GraphQL, Microservices, and Serverless, we still cannot get services right - did you wonder why? IMHO it's all about principles and isolation is at the CORE of everything. Software will get more complex at Scale, and change will always happen. The question what are you doing to be more anti-fragile? Proper services require thinking and proper architecture and design principles applied at all times during a service SDLC.

Software engineering keeps changing names for the same old problems. Before 2023, changes were introduced with some level os stability; now, changes happen much faster, to the point that we do not really understand the problems. We had those problems in the past, considering things like (agile/scrum, microservices,  devops), but never has it been a fast loop like this. First, we had scripting. Then automation. Then CI/CD. Then DevOps. Then Platform Engineering. Then AI Agents. Now we have Loop Engineering. However, companies have barely functional CI/CD due to a lack of incentives, poor vision, poor management, and other dysfunctions.  Loop engineering, the name is new. The problem is not. The problem is: how do you make a machine do useful engineering work without babysitting every step? That’s it. But "solution" and "waste" manifest in the same way, using the same tools. Loop Engineering is not just about prompting with a fancy jacket. It is not “write a bette...

DevOps Engineering is often "perceived" as a operations matter. In fact, it's true but is not 100% JUST ops. There are some blurred lines between engineering and operations. When we take a look at the DevOps ecosystem we can highlight some great tools like Ansible, Terraform, Jenkins, Packer and so many others. Provisioning, Release, Observability, and SRE are different aspects of a broader DevOps adoption. The higher benefit comes when we start applying more engineering to DevOps. It's very important to have good tooling around. When we think about tooling engineering makes a lot of sense, however, even outside of the tooling space there are areas where we definitely need an engineering injection like Cloud Production Operation Automation. Releasing is the easy part, things get complicated when you automate the operational aspects of your systems. It sounds so meta or Inception(If you watch the movie ) but Systems managing Systems looks the way to go for me. This ...

Refactoring is so underrated and misunderstood. There will never be enough books or enough blog posts about it. It's something that we need to keep revisiting from time to time. Very few people deeply understand it, but one thing is for sure: genuine and spread across the whole industry like wildfire: fear of refactoring. Fear can be a powerful feeling and an excellent way to keep you in check with reality, especially in difficult economic times. Still, fear can also be a significant source of blindness, inertia, and illusion, leading to bad economic decisions. Refactoring is not a binary decision; you either do it all and re-write the system or do zero-to-nothing. Companies and engineers need to apply more refactoring daily , and this matter is definitely unbalanced. Software at scale makes complete rewrites almost impossible. No company can stop the business for years and still succeed; many companies have tried and failed financially and in rewriting. So don't do more refact...

Services are a very important architectural construct. Proper services( POSAM definition and principles) are even more important. However, not all services are the same. Not all APIs are the same, and definitely not all services are the same. Just having a Service or API, we can call it many different ways; the question is how good it is and how solid it is. I still put my money where my mouth is and say that services are a very powerful concept and the right level of abstraction for most problems. However, when a service is presented to you, you cannot assume you will have high quality just because it's a service. IF services are desired, multiple services must still be desired; after all, service orientation is about having services, not libraries. However, we know that when the granularity is wrong, and when we move data and capability to the wrong places, let's say, the wrong domain, we make things worse, not only risking breaking isolation but also leading to wrong abstra...