Thursday, November 6, 2014

Dig a Little Deeper

Be sure to separate symptoms of a problem from the problem itself.  This can sometimes be harder than it seems.  Sometimes the problem is a few layers below the surface.  For example, assume you get the following issue reports and feature requests:
1.     Please add a button to all list screens to export data to Excel.
2.     Whenever running a query that returns a lot of data (e.g. 10K rows) the list screens take a long time to return.
3.     We’d like to be able to run very basic (read-only) queries by linking to some tables through MS Access.  But in order to do that we need an Oracle client installed on our machine.  We also need a username/password for the database.
4.     Please add “week ending” and “month” to the return fields for all list screens.
On the surface these seem like simple, disparate problems to be solved.  So you might come up with solutions like this:
1.     “Sure – we can add an export button”.
2.     “Hmmm – we’ll have to look at the query – maybe it can be tuned.”
3.     “A little unorthodox.  We’ll have to get approval from IT management that this is ok but this is not technically challenging”.
4.     “Sure – easy enough”
But if we look a little deeper we can see that these are symptoms of a deeper problem: the users do not have an easy way to perform management reporting and data analysis.  Neither this system nor other systems (e.g. a data warehouse) are addressing these business needs.  Think about it.  Here is a different way of looking at each of the issues:
1.     Adding an export button is fairly harmless but the reason they need it is probably because they are doing analysis in Excel that the system is not helping them with.
2.     There’s no way they’re reading through 10K lines of data line-by-line.  The reason they’re doing this is probably because they’re analyzing the data in ways that the system does not support.
3.     There’s no way they’d be desperate enough to navigate the oddities of our back-end if the system better supported their analytical needs.
4.     Hmmm.  I am guessing they want those fields for time-trend reporting or filtering.  This suggests that they’re doing their analysis off of large report results which is another symptom of the fact that the system does not support their analytical needs.
Whenever possible focus on ideas to treat the problem not the symptoms.  Treating symptoms is ok as a short-term alternative but it is rarely the way to go in the long-term.  To use a medical analogy cough syrup is useful and may effectively treat the symptoms of an infection but antibiotics have the capability to address the root cause.  Dig a little deeper and be sure you get to the underlying problem.

Thursday, September 18, 2014

A Response to the Reactive Manifesto

The Reactive Manifesto published by Typesafe is getting a great deal of attention in the Scala community.  A new version was posted yesterday and I could not help but reply with criticism that has been bothering me for some time.  The Reactive Manifesto appears to be a thinly veiled way to back into Akka being the cure for all ills.  I think this is an irresponsible message especially from a company like Typesafe that is guiding a great number of people that are just venturing into Scala.  One sure way to hurt the Scala community is to offer the advice that one should adhere to a reference architecture that is best applied for only a certain set of problems.  My comment/reply to the Manifesto appears below in its entirety.  I hope that, at a minimum, it stimulates some thought for those that were intending to jump into the Akka pool without fully understanding why.


I am the VP of Engineering at LeadiD a tech startup based in Ambler, PA. We are building out a large, complex stack for servicing web transactions at scale. We already process billions of web service calls each month. We are gung-ho on Scala even though it’s new to us here. But I must confess that so far I’ve felt the Reactive Manifesto to be a bit, if you’ll forgive me, contrived. When I read it, it seems that we’ve all been suffering from ADD (Akka Deficit Disorder) and we just didn’t know it. But actors (which are at the heart of Akka) are not the cure for everything. They are a great solution for certain types of problems.

Allow me to comment on the system attributes in the Manifesto:

Responsive - An essential attribute. This is a function of proper provisioning. There are many ways to achieve this.

Resilient - An essential attribute. Most mission-critical systems ensure this using a load balancer and a cluster(s) of servers. Does not require the use of actors or supervision hierarchies.

Elastic - An essential attribute. Akka does have a sweet spot here. But can be achieved using elastic provisioning techniques by rolling your own in a DevOps manner or by using third party tools.

Message-Driven - This is not an essential attribute. You’ve fast-forwarded to a solution, here. It’s a means to an end and it basically describes actors. Increasing asynch is very important as a piece of the puzzle. But synchronous processing of a request that does not lend itself to being split up or parallelized still has its place.

There are some complexities with actors:
  • Actors are a mechanism to solve a distributed computing problem - that is, when a given request is best split up across a number of machines. When a given problem does not lend itself to distributed computing it’s not a great fit. Many CRUD functions (even at scale) out there are served best by a cluster of nearly identical, load-balanced app servers each of which is self-contained.
  • Unless you have a distributed computing problem location transparency is not a good thing. I want the caller to know he’s making a network call vs. an in memory lookup.
  • Given the buzz people will gravitate to actors because it appears to be the latest / greatest. But unless their problem merits it they’ve opened up new cans of worms. If the work can be handled in a single process actors might not be the right choice.
Rich Hickey says it best:

“I chose not to use the Erlang-style actor model for same-process state management in Clojure for several reasons:
  • It is a much more complex programming model, requiring 2-message conversations for the simplest data reads, and forcing the use of blocking message receives, which introduce the potential for deadlock. Programming for the failure modes of distribution means utilizing timeouts etc. It causes a bifurcation of the program protocols, some of which are represented by functions and others by the values of messages.
  • It doesn't let you fully leverage the efficiencies of being in the same process. It is quite possible to efficiently directly share a large immutable data structure between threads, but the actor model forces intervening conversations and, potentially, copying. Reads and writes get serialized and block each other, etc.
  • It reduces your flexibility in modeling - this is a world in which everyone sits in a windowless room and communicates only by mail. Programs are decomposed as piles of blocking switch statements. You can only handle messages you anticipated receiving. Coordinating activities involving multiple actors is very difficult. You can't observe anything without its cooperation/coordination - making ad-hoc reporting or analysis impossible, instead forcing every actor to participate in each protocol.
  • It is often the case that taking something that works well locally and transparently distributing it doesn't work out - the conversation granularity is too chatty or the message payloads are too large or the failure modes change the optimal work partitioning, i.e. transparent distribution isn't transparent and the code has to change anyway.”
And then there is Fowler’s First Law of Distributed Object Design: Don’t distribute your objects! (unless you must)

Typesafe is shepherding a great number of people in the Scala community on how to best use the language and its platform. By pushing Reactive (which basically equals Akka) on its front page as the cure for all ills it suggests that you must be missing something if you’re not doing it this way. But the way that’s been outlined is most appropriate in a distributed computing setup. And many distributed computing problems arising out of large datasets, for example, can be addressed using other techniques such as MapReduce.

While the Akka platform is very powerful I believe that the Scala community could use better education from Typesafe on when it is best used as well as when to consider alternative architectures. Balance will go a long way toward credibility. Thanks so much for the work. But please bear in mind that if Akka is a hammer that does not imply that everything is a nail.

Joe Lynch

Tuesday, April 8, 2014

Build Upon Rock: Understand the Business Need

Because software developers are in the business of delivering solutions our thinking and discussions with business owners tend to be solution-oriented. It’s not uncommon to hear things like this from business owners:
  • “I really need a new column added to the F105 report – the total processing time.”
  • “I need a new screen that tracks inventory.”

It’s also not uncommon for developers to listen to business owners talk about their issues and jump to well-known solutions and say things like:
  • “What you need is a CRM system.”
  • “The F109 screen will take of everything you need. I’ll show you how to use it.”

Thinking about solutions is not a bad thing but it can lead to mistakes if the business problem is not framed properly. You can tell that you’ve framed a business problem properly when its description has nothing to do with a system, per se. Take the case of adding a column to the F105 report. Rather than respond – “ok – we’ll get right on that,” start by getting to the underlying business need:

Developer: “What's changed? What is motivating the need to add the total processing time to the report?”

Business Owner: “We’ve changed our processes so that we’re reporting on this to our customers. Upper management is also really hot on it – we had some very slow fulfillment times last month.”

Developer: “Oh. Ok. It’s easy to add it to the report but that’s just a lagging indicator. Do you think you might need tools that alert you to orders that haven’t been fulfilled in a certain amount of time? Then you’d have a chance to head the issues off before they reach a certain point of criticality. So rather than just a passive indicator we could give you a proactive tool for managing exceptions.”

Business Owner: “Yeah. That’s even better.”

If you understand the underlying business need you will be able to offer alternative solutions and improve upon solutions that may have already been suggested. You’ll be that much more valuable and seen as an architect of solutions, not just someone who implements things blindly without understanding implications. Moreover, understanding the underlying business needs will teach you more and more about the business that you are supporting which will make you far more effective in your work. You’ll be able to make logical deductions based on business understanding that would otherwise not be obvious. And you’ll become the go-to person when your business owners are facing a tough problem – business owners love people that can speak their language and that can help bridge the gap between what they think they need and what they really need.

Related to understanding business needs, it is imperative to ensure that one has framed a problem properly before beginning to work on it. An inaccurate framing of a problem can lead to serious mistakes not the least of which is that one may be focusing on solving the wrong problem. As human beings, our brains are conditioned through millions of years of evolution to see patterns in order for us to process information quickly – we are wired to put things into categories and react accordingly as this has been necessary for the survival of the species. For example, if one perceives a threat (e.g. a hissing cobra) the amygdala takes over and ensures that we react immediately via a fight or flight response. The amygdala is an ancient part of the brain which is not used to solve a difficult math problem – a different portion of the brain (the frontal lobe) is used to solve that problem. Our biology itself is pre-conditioned for pattern-recognition.

As seers of patterns, it is easy to leap to conclusions in either the definition of the problem or worse, its solution. We sometimes make things fit within a box. Sometimes we may even reverse-engineer a problem to fit a pre-conceived solution! Think about it. “I’ve got a meeting with the head of the Customer Service team. Sounds like he needs a CRM.” Then you go to the meeting and due to the human tendency toward confirmation bias all you hear are the issues that a CRM could help solve. “See – told you he needed a CRM.” But you walk away not hearing the fact that this person is dealing with other issues that you might have been able to help with. And after all, have we even framed the problem? What is the problem? CRM-Deficit-Disorder?

Imagine the following scenario. A business owner comes running into your office. “I can’t run the F105 report! I enter the search criteria and then the application just crashes. I need to get these numbers to our customer within an hour. Help!” Being a diligent production support team member you make sure you can replicate the issue. Good – you can. You start seeing if you can fix the crash. You queue up the testing and deployment staff – “hey – we’ve got a hot bug fix coming – be ready”. 45 minutes goes by and you’re still having trouble figuring out why it’s crashing. The business comes in very concerned. “Well?” “I’m working on it” you say. 15 minutes go by and then 30 minutes and then a few hours. By the time you get the issue fixed it’s 7:00 PM. Your business owner says – “Since we didn’t get it out by 4 PM the customer’s going to be pretty angry. But thanks for getting it done as quickly as you could. I know you tried.” You say to yourself – “I did the best I could but I only had a few hours' notice. Oh well.” You go home feeling pretty good about yourself. You kiss the kids goodnight and hold your head up high.

But did you do a good job? What was the problem? You might think – “Duh! Clearly the problem was that there was a bug in the F105 report that was preventing it from being run”. Wrong! The immediate problem was that the business owner was not able to submit required information to the customer by 4:00 thus causing the company to fail in the customer’s eyes. Framed that way, one obvious solution would have been to run the query behind the report (manually) and give the business owner the results. But business owners are not always aware of options available to them – we have to help flesh them out – most have no idea how a system is put together. Framing a problem properly can lead to a broader solution space and can lead one to the best solution faster. Framing a problem incorrectly can, in some cases, lead to disaster.

Understand the business need, frame the problem properly and your solutions will be built upon rock.

Thursday, March 27, 2014

Software Development Methodology Patterns

“I promise not to exclude from consideration any idea based on its source, but to consider ideas across schools and heritages in order to find the ones that best suit the current situation.”
-- The Oath of Non-Allegiance, Alistair Cockburn, Co-Creator of the Agile Manifesto

One of my current interests is enumeration of common patterns within software development methodologies that might become part of a standard toolkit. I do not assume that all patterns apply equally well in every circumstance. I reject the idea that there is a silver-bullet methodology that will work for all types of projects – even considering that one might be defining patterns to be used at a single company with a single culture. The idea that strict methodologies apply well to all projects in all companies I reject with even more vehemence.

So if each situation has some uniqueness, should we reject methodology altogether? Of course not. But rather than assume a monolithic prescriptive methodology can succeed, the methodology can be itself a fabric of patterns. Some will be used in all situations, some followed in most and some followed only under special circumstances.

Why use the term “pattern”? Like the concept of a “design pattern” made popular by the Gang of Four, the patterns apply only in context. Just as one would not use the Observer pattern without a context that merited it, the same can apply for an aspect of the development process such as the creation of a technical design document. Might it make sense to create a technical design document for a new, complex system interface? Absolutely. Might it make sense to create a technical design document for adding a new, clearly-defined field to an existing report? No – this is a perfect example of documentation for documentation’s sake – a common source of waste that both Agile and Lean methodologies eschew.

Most of the things that I’ve seen work are borrowed from established methodologies such as RUP, Scrum, XP, Lean and Kanban. I have no interest in an ivory tower methodology that somehow purports to be the new, unique way to develop software, period. I make every effort to include those portions of existing methodologies that I've seen work and leave behind those that I have seen fail. Beyond the contributions of formal methodologies, patterns can and should also be infused with tribal wisdom and collective experience – things that just work, regardless of whether you’ll ever read them in a book. By forming a library of patterns we can create the ability to define methodologies dynamically based on the class of project at hand or even based on the needs of individual projects themselves. This is a simple but powerful idea.

I plan to wrestle with this theme (among others) in future blog entries. And like all good ideas, this one came from someone else – it is well-articulated in SDLC 3.0, a fabulous book by Mark Kennaley, which I highly recommend.