Wednesday, April 6, 2016

A Conversation That Changed My Life

I usually post tech things on my blog but before being a technologist, I'm actually a human and occasionally experience something worth sharing. And so, an anecdote...

In the fall of 2013 I found myself exploring new career opportunities.  At the time, my network within the Philly community was fairly weak as I had worked in Princeton, NJ.  I was targeting VP-level Technology opportunities in the Philly area but I had no idea where to start.  I met a recruiter who had a position open.  We met over coffee and it turned out I wasn't a fit.  But he introduced me to a man that he used to work with whom he said "helps people out".

This man didn't know me from Adam.  But on fairly short notice he agreed to talk with me over the phone on a Friday evening.  He had a long drive in front of him and we ended up talking for about an hour and a half.  He listened to me and made me feel comfortable.  He didn't tell me I was aiming too high like others did.  He anticipated my needs.  He talked to me about all kinds of things: networking groups, how to find opportunities and "paperwork", as he called it.  Turns out he too was on the lookout for new opportunities but he only mentioned it in passing.  He was commuting hundreds of miles each week for an out-of-state CIO position.  He would spend the weekdays at work and then come home on the weekends.  I tried to pin him down for a cup of coffee but he only suggested that maybe he'd swing by my town one day as he liked to drive along the Delaware river.  At the end of the call he said he'd send me some stuff when he got home.  A few hours later I received ~6 emails, each of which had zip files with several Megabytes worth of attachments that had been curated and collected from the internet - resumes, cover letters, recruiting firms, bonuses, stock options, negotiating comp.  It looked like it was the result of ~3 months of active research.  In the email trail I could see that he had forwarded these emails to dozens of people.  All he did was re-send them from his Sent folder.  Without prompting, he also introduced me to a few people over email, including his brother.  (Who the hell introduces their brother to someone that they just met over the phone?)  I had no idea what to make of this.  I am naturally suspicious of people that want to "help" with no apparent motive.  Over the coming months he introduced me to several other great contacts.  And I managed to put him in touch with one or two people that might have a good job for him, in order to assuage my sense of guilt.  Yes, guilt.  I needed to repay him but how could I?  He didn't seem to want anything in return.

Since that time, I have become a more active networker.  From his example the only trick I've learned when it comes to networking is to just try to help people.  That's it.  Nothing more, nothing less.  Ask "How can I help?" and mean it.  Since meeting him, I read a book called "Influence: The Psychology of Persuasion" by Cialdini.  After reading it (in addition to feeling like an idiot for having bought an unsolicited meat freezer and half a cow from a salesman who took me for a ride) I started to understand my feelings.  Built into our brains is the unwritten rule of reciprocity.  Our society has functioned best when humans both give and take.  But we don't like people that take exclusively and we sniff them out and shun them.  I'll give you a hunk of meat around the campfire if I can expect that when I'm in a pinch you might give me a hunk of yours.  Our biology has evolved this function to support the survival of the species.  But this was different.  This man was a giver and didn't look for anything in return - he gave away his meat without keeping tabs.  My brain's instinct toward reciprocity caused me to want to pay it forward to others if I couldn't find a way to repay him directly.  At least then my debits and credits would balance out.  But I found out that it doesn't really work that way.  Once I got used to (occasionally) helping people I found it rewarding.  I got to play a small part in their success and to feel a tiny hint of meaning amidst the bustle of everyday work.

I recently met him in person at a networking luncheon.  I was so thrilled!  I reminded him that he had helped me and that I was grateful.  It rolled off him like water off the back of a duck.  "That's great" he said and then moved on to something else.  He was no master of social etiquette.  The woman who led the group (a high priestess of networking herself) said "keep so-and-so in your thoughts as he's sick" and he blurted out "yeah - he has cancer!".  He was promptly scolded (in good humor) by the group leader: "hey - we're trying to keep the details private!".  I was looking forward to talking to him after the event but he darted out as soon as it was over.  "Oh well - I'll get him next time" I thought.

A few weeks ago I was scheduled to have coffee with his brother, the same one that he had introduced me to.  He's another active networker that is generous with his time.  He cancelled the day before.  He learned that his brother had died suddenly that morning.  In the email, sent on the day that he'd learned of his brother's unexpected passing, he suggested alternate dates when we could meet.  I couldn't bring myself to re-schedule.

I barely got a chance to say hello let alone a chance to say goodbye.  But an hour and a half on the phone changed the way that I think about professional relationships.  His name?  His name was Joe Tait and I suspect that he helped hundreds of people the way that he helped me.  Thanks, Joe.  Still looking for a way to pay you back...



Friday, July 24, 2015

Non-Production Environments: Anti-Patterns

Introduction

 

We all know that clean, separated environments are critical for developing and testing our software.  And yet somehow we tend to get it wrong, often.  The Continuous Delivery movement puts this at the forefront of its concerns, which is great.  So if you're headed down the CD path, chances are you are tackling or are about to tackle the "environment problem".  Because it's a tough thing to get right and because it involves up-front and continuous investment, it tends to be de-emphasized in the feature-crazed Agile world.  But unless you're just doing a proof-of-concept, not solving the environment problem will slow you down, hurt quality and become harder to fix over time.  There are a few anti-patterns that I see over and over:
  1. Not giving proper attention to the data tier(s) of the stack (e.g. relational database, NoSQL database, text/search database, file systems used for storage, etc.)
  2. Not extending environments "all the way out".  You should be able to have environments that go up and down: CDN->Load Balancer->Web Server->App Server(s)->DB Server(s), etc.
  3. Assuming that the number of non-production environments that you need is fixed and can be predetermined.
So, let's talk a little about these a bit, one at a time...

Not giving proper attention to the data tier(s) of the stack

 

The reason this gets skimped on is because it's hard!  Some issues:
  • ORMs have encouraged ignorance of relational databases and people don't work hard at things they do not understand.  If you don't understand a relational database, what are the chances you're going to work at getting a synch / replication / sub-setting process built?  Most developers who have come up in the last 10 years have a small fraction of understanding of relational databases when compared to those who came up before that.  Before ORMs, you had to understand relational databases in detail to build a complex application.  ORMs have been both a blessing and a curse in some ways.  Many would say only a curse.  One of the "curse" aspects is that they encourage ignorance where ignorance is to our detriment.  Despite the emergence of other data stores, relational databases will survive for many years for certain use cases.  The ACID and strong consistency guarantees are very convenient for certain problems.  If distributed ACID emerges (e.g. FoundationDb, NuoDb attempt this) then that could move some of those use cases away from the relational model.  But most (balanced) experts agree that for the foreseeable future, relational databases remain an essential aspect of storage even as NoSQL takes over certain use cases.  One notable exception is Michael Stonebraker, inventor of PostgreSQL, VoltDb, recent Turing award winner and certainly smarter man than me.  He predicts that column stores will obviate relational databases within the data warehouse arena and that distributed ACID systems like VoltDb will take over what remains of their share in the OLTP market.  However VoltDb is an in-memory database and he doesn't (as far as I can tell) talk about distributed ACID that's too big to fit in memory.  That's what FoundationDb, etc. are trying to address.  He is a guru but also extreme in his views.  I agree that relational databases can (and perhaps will) be completely overtaken - but distributed ACID large-scale databases need to be built for that to happen.  It's interesting to see that Google has moved away from BigTable and built a distributed ACID system for its developers. (Spanner, F1)  Unfortunately it depends upon specialized hardware at the data center (atomic & GPS clocks) that is not yet available to the average consumer of the public cloud.  Maybe CaaS (clocks as a service) will be added by AWS someday.
  • Increasingly, production systems are bigger and bigger.  So having any production-like data means having to copy around TBs of data.  To get it right you must often subset the data.  Both dealing with TBs and sub-setting TBs is hard work.  Sub-setting, for example, requires refinement and continuous maintenance as you add new types of data.  You can't approach the problem generically.
  • Given the explosion of database options ("polyglot persistence") it's common to have disparate types of databases for different use cases.  Each of these has to be well-understood to have good data environments.  This brings complexity.  For example, at LeadiD, we like Couchbase for a doc store and separately for a cache/key-value store, Elasticsearch for text/search and logging, PostgreSQL for relational, Azure and S3 for file storage, Kafka for messaging and Graphite for metrics.  That's a lot of things to get right and we're fairly careful about bringing in another data store.

 

Not extending environments "all the way out"

 

An environment should extend across all layers and tiers of a system.  For example, the CDN tier, the load-balancer tier, web server tier, app server(s) tier, data tiers, etc.  If you have access to a cloud infrastructure, this is fairly straightforward.  It just takes foresight and discipline.  With REST APIs for everything needed such as DNS management, CDN management, virtualization, Docker, etc., the tools are there to get this right.  It requires discipline but is nowhere near as hard as getting the data stack right.

 

Assuming that the number of non production environments that you need is fixed

 

You cannot know the number of non-production environments you need in advance and yet people make the big mistake of hard-coding and only configuring 3-4 environments total (e.g. Dev/QA/UAT, Dev/QA/UAT/Staging, Dev/QA/Staging).  There are two problems with this:
  1. You need one of these for every independent code path (e.g. might be a sub-team or a long-lived branch [sometimes you have no choice])
  2. You need more than these for some code paths - and you can't always predict which ones you'll need.  For example, it's common to have a "performance" environment that has enough data to tune nasty queries.  It's common to have a "prod repro" environment that has a continually-upkept database (maybe nightly) to be able to reproduce transaction-specific bugs that were recently reported.  There are environments to deal with automated testing, environments to practice data loads and environments for a particularly invasive feature a developer might be working on.  You cannot predict the number of independent code paths.  And you cannot predict the number of environments you'll need to function optimally along a given code path.
One manifestation of this cardinal sin is code (even if test code) that hard-codes understanding of fixed non-production environments or assumes that only certain of these exist.  (e.g. "if (env == 'QA') then ...")  This is bad.  Ideally your code should be completely environment-agnostic.  If your back's up against the wall you can code for detecting production vs. non-production.  But that's it.

 

Outlook

 

While every setting I've worked in has had variations of these problems I have tried hard to at least deprecate these practices and make headway toward reversing them.  But of course there is no such thing as true greenfield unless you're employee number 1.  And if you're employee number 1 it's irresponsible to be spending this much time thinking about these things because who knows if you even have a product?  And so it comes down to knowing up-front what best practices are, accepting that in the beginning you'll incur some tech debt but moving to good environment practices as soon as you can tell that you have a product.  At LeadiD we have many of these problems just like anyone else.  We're tackling them now as we embark upon the journey of Continuous Delivery.  Fingers crossed...

Thursday, November 6, 2014

Dig a Little Deeper


Be sure to separate symptoms of a problem from the problem itself.  This can sometimes be harder than it seems.  Sometimes the problem is a few layers below the surface.  For example, assume you get the following issue reports and feature requests:
1.     Please add a button to all list screens to export data to Excel.
2.     Whenever running a query that returns a lot of data (e.g. 10K rows) the list screens take a long time to return.
3.     We’d like to be able to run very basic (read-only) queries by linking to some tables through MS Access.  But in order to do that we need an Oracle client installed on our machine.  We also need a username/password for the database.
4.     Please add “week ending” and “month” to the return fields for all list screens.
On the surface these seem like simple, disparate problems to be solved.  So you might come up with solutions like this:
1.     “Sure – we can add an export button”.
2.     “Hmmm – we’ll have to look at the query – maybe it can be tuned.”
3.     “A little unorthodox.  We’ll have to get approval from IT management that this is ok but this is not technically challenging”.
4.     “Sure – easy enough”
But if we look a little deeper we can see that these are symptoms of a deeper problem: the users do not have an easy way to perform management reporting and data analysis.  Neither this system nor other systems (e.g. a data warehouse) are addressing these business needs.  Think about it.  Here is a different way of looking at each of the issues:
1.     Adding an export button is fairly harmless but the reason they need it is probably because they are doing analysis in Excel that the system is not helping them with.
2.     There’s no way they’re reading through 10K lines of data line-by-line.  The reason they’re doing this is probably because they’re analyzing the data in ways that the system does not support.
3.     There’s no way they’d be desperate enough to navigate the oddities of our back-end if the system better supported their analytical needs.
4.     Hmmm.  I am guessing they want those fields for time-trend reporting or filtering.  This suggests that they’re doing their analysis off of large report results which is another symptom of the fact that the system does not support their analytical needs.
Whenever possible focus on ideas to treat the problem not the symptoms.  Treating symptoms is ok as a short-term alternative but it is rarely the way to go in the long-term.  To use a medical analogy cough syrup is useful and may effectively treat the symptoms of an infection but antibiotics have the capability to address the root cause.  Dig a little deeper and be sure you get to the underlying problem.

Thursday, September 18, 2014

A Response to the Reactive Manifesto

The Reactive Manifesto published by Typesafe is getting a great deal of attention in the Scala community.  A new version was posted yesterday and I could not help but reply with criticism that has been bothering me for some time.  The Reactive Manifesto appears to be a thinly veiled way to back into Akka being the cure for all ills.  I think this is an irresponsible message especially from a company like Typesafe that is guiding a great number of people that are just venturing into Scala.  One sure way to hurt the Scala community is to offer the advice that one should adhere to a reference architecture that is best applied for only a certain set of problems.  My comment/reply to the Manifesto appears below in its entirety.  I hope that, at a minimum, it stimulates some thought for those that were intending to jump into the Akka pool without fully understanding why.

Jonas:

I am the VP of Engineering at LeadiD a tech startup based in Ambler, PA. We are building out a large, complex stack for servicing web transactions at scale. We already process billions of web service calls each month. We are gung-ho on Scala even though it’s new to us here. But I must confess that so far I’ve felt the Reactive Manifesto to be a bit, if you’ll forgive me, contrived. When I read it, it seems that we’ve all been suffering from ADD (Akka Deficit Disorder) and we just didn’t know it. But actors (which are at the heart of Akka) are not the cure for everything. They are a great solution for certain types of problems.

Allow me to comment on the system attributes in the Manifesto:

Responsive - An essential attribute. This is a function of proper provisioning. There are many ways to achieve this.

Resilient - An essential attribute. Most mission-critical systems ensure this using a load balancer and a cluster(s) of servers. Does not require the use of actors or supervision hierarchies.

Elastic - An essential attribute. Akka does have a sweet spot here. But can be achieved using elastic provisioning techniques by rolling your own in a DevOps manner or by using third party tools.

Message-Driven - This is not an essential attribute. You’ve fast-forwarded to a solution, here. It’s a means to an end and it basically describes actors. Increasing asynch is very important as a piece of the puzzle. But synchronous processing of a request that does not lend itself to being split up or parallelized still has its place.

There are some complexities with actors:
  • Actors are a mechanism to solve a distributed computing problem - that is, when a given request is best split up across a number of machines. When a given problem does not lend itself to distributed computing it’s not a great fit. Many CRUD functions (even at scale) out there are served best by a cluster of nearly identical, load-balanced app servers each of which is self-contained.
  • Unless you have a distributed computing problem location transparency is not a good thing. I want the caller to know he’s making a network call vs. an in memory lookup.
  • Given the buzz people will gravitate to actors because it appears to be the latest / greatest. But unless their problem merits it they’ve opened up new cans of worms. If the work can be handled in a single process actors might not be the right choice.
Rich Hickey says it best:

“I chose not to use the Erlang-style actor model for same-process state management in Clojure for several reasons:
  • It is a much more complex programming model, requiring 2-message conversations for the simplest data reads, and forcing the use of blocking message receives, which introduce the potential for deadlock. Programming for the failure modes of distribution means utilizing timeouts etc. It causes a bifurcation of the program protocols, some of which are represented by functions and others by the values of messages.
  • It doesn't let you fully leverage the efficiencies of being in the same process. It is quite possible to efficiently directly share a large immutable data structure between threads, but the actor model forces intervening conversations and, potentially, copying. Reads and writes get serialized and block each other, etc.
  • It reduces your flexibility in modeling - this is a world in which everyone sits in a windowless room and communicates only by mail. Programs are decomposed as piles of blocking switch statements. You can only handle messages you anticipated receiving. Coordinating activities involving multiple actors is very difficult. You can't observe anything without its cooperation/coordination - making ad-hoc reporting or analysis impossible, instead forcing every actor to participate in each protocol.
  • It is often the case that taking something that works well locally and transparently distributing it doesn't work out - the conversation granularity is too chatty or the message payloads are too large or the failure modes change the optimal work partitioning, i.e. transparent distribution isn't transparent and the code has to change anyway.”
And then there is Fowler’s First Law of Distributed Object Design: Don’t distribute your objects! (unless you must)

Typesafe is shepherding a great number of people in the Scala community on how to best use the language and its platform. By pushing Reactive (which basically equals Akka) on its front page as the cure for all ills it suggests that you must be missing something if you’re not doing it this way. But the way that’s been outlined is most appropriate in a distributed computing setup. And many distributed computing problems arising out of large datasets, for example, can be addressed using other techniques such as MapReduce.

While the Akka platform is very powerful I believe that the Scala community could use better education from Typesafe on when it is best used as well as when to consider alternative architectures. Balance will go a long way toward credibility. Thanks so much for the work. But please bear in mind that if Akka is a hammer that does not imply that everything is a nail.

Joe Lynch





Tuesday, April 8, 2014

Build Upon Rock: Understand the Business Need

Because software developers are in the business of delivering solutions our thinking and discussions with business owners tend to be solution-oriented. It’s not uncommon to hear things like this from business owners:
  • “I really need a new column added to the F105 report – the total processing time.”
  • “I need a new screen that tracks inventory.”

It’s also not uncommon for developers to listen to business owners talk about their issues and jump to well-known solutions and say things like:
  • “What you need is a CRM system.”
  • “The F109 screen will take of everything you need. I’ll show you how to use it.”

Thinking about solutions is not a bad thing but it can lead to mistakes if the business problem is not framed properly. You can tell that you’ve framed a business problem properly when its description has nothing to do with a system, per se. Take the case of adding a column to the F105 report. Rather than respond – “ok – we’ll get right on that,” start by getting to the underlying business need:

Developer: “What's changed? What is motivating the need to add the total processing time to the report?”

Business Owner: “We’ve changed our processes so that we’re reporting on this to our customers. Upper management is also really hot on it – we had some very slow fulfillment times last month.”

Developer: “Oh. Ok. It’s easy to add it to the report but that’s just a lagging indicator. Do you think you might need tools that alert you to orders that haven’t been fulfilled in a certain amount of time? Then you’d have a chance to head the issues off before they reach a certain point of criticality. So rather than just a passive indicator we could give you a proactive tool for managing exceptions.”

Business Owner: “Yeah. That’s even better.”

If you understand the underlying business need you will be able to offer alternative solutions and improve upon solutions that may have already been suggested. You’ll be that much more valuable and seen as an architect of solutions, not just someone who implements things blindly without understanding implications. Moreover, understanding the underlying business needs will teach you more and more about the business that you are supporting which will make you far more effective in your work. You’ll be able to make logical deductions based on business understanding that would otherwise not be obvious. And you’ll become the go-to person when your business owners are facing a tough problem – business owners love people that can speak their language and that can help bridge the gap between what they think they need and what they really need.

Related to understanding business needs, it is imperative to ensure that one has framed a problem properly before beginning to work on it. An inaccurate framing of a problem can lead to serious mistakes not the least of which is that one may be focusing on solving the wrong problem. As human beings, our brains are conditioned through millions of years of evolution to see patterns in order for us to process information quickly – we are wired to put things into categories and react accordingly as this has been necessary for the survival of the species. For example, if one perceives a threat (e.g. a hissing cobra) the amygdala takes over and ensures that we react immediately via a fight or flight response. The amygdala is an ancient part of the brain which is not used to solve a difficult math problem – a different portion of the brain (the frontal lobe) is used to solve that problem. Our biology itself is pre-conditioned for pattern-recognition.

As seers of patterns, it is easy to leap to conclusions in either the definition of the problem or worse, its solution. We sometimes make things fit within a box. Sometimes we may even reverse-engineer a problem to fit a pre-conceived solution! Think about it. “I’ve got a meeting with the head of the Customer Service team. Sounds like he needs a CRM.” Then you go to the meeting and due to the human tendency toward confirmation bias all you hear are the issues that a CRM could help solve. “See – told you he needed a CRM.” But you walk away not hearing the fact that this person is dealing with other issues that you might have been able to help with. And after all, have we even framed the problem? What is the problem? CRM-Deficit-Disorder?

Imagine the following scenario. A business owner comes running into your office. “I can’t run the F105 report! I enter the search criteria and then the application just crashes. I need to get these numbers to our customer within an hour. Help!” Being a diligent production support team member you make sure you can replicate the issue. Good – you can. You start seeing if you can fix the crash. You queue up the testing and deployment staff – “hey – we’ve got a hot bug fix coming – be ready”. 45 minutes goes by and you’re still having trouble figuring out why it’s crashing. The business comes in very concerned. “Well?” “I’m working on it” you say. 15 minutes go by and then 30 minutes and then a few hours. By the time you get the issue fixed it’s 7:00 PM. Your business owner says – “Since we didn’t get it out by 4 PM the customer’s going to be pretty angry. But thanks for getting it done as quickly as you could. I know you tried.” You say to yourself – “I did the best I could but I only had a few hours' notice. Oh well.” You go home feeling pretty good about yourself. You kiss the kids goodnight and hold your head up high.

But did you do a good job? What was the problem? You might think – “Duh! Clearly the problem was that there was a bug in the F105 report that was preventing it from being run”. Wrong! The immediate problem was that the business owner was not able to submit required information to the customer by 4:00 thus causing the company to fail in the customer’s eyes. Framed that way, one obvious solution would have been to run the query behind the report (manually) and give the business owner the results. But business owners are not always aware of options available to them – we have to help flesh them out – most have no idea how a system is put together. Framing a problem properly can lead to a broader solution space and can lead one to the best solution faster. Framing a problem incorrectly can, in some cases, lead to disaster.

Understand the business need, frame the problem properly and your solutions will be built upon rock.

Thursday, March 27, 2014

Software Development Methodology Patterns

“I promise not to exclude from consideration any idea based on its source, but to consider ideas across schools and heritages in order to find the ones that best suit the current situation.”
-- The Oath of Non-Allegiance, Alistair Cockburn, Co-Creator of the Agile Manifesto

One of my current interests is enumeration of common patterns within software development methodologies that might become part of a standard toolkit. I do not assume that all patterns apply equally well in every circumstance. I reject the idea that there is a silver-bullet methodology that will work for all types of projects – even considering that one might be defining patterns to be used at a single company with a single culture. The idea that strict methodologies apply well to all projects in all companies I reject with even more vehemence.

So if each situation has some uniqueness, should we reject methodology altogether? Of course not. But rather than assume a monolithic prescriptive methodology can succeed, the methodology can be itself a fabric of patterns. Some will be used in all situations, some followed in most and some followed only under special circumstances.

Why use the term “pattern”? Like the concept of a “design pattern” made popular by the Gang of Four, the patterns apply only in context. Just as one would not use the Observer pattern without a context that merited it, the same can apply for an aspect of the development process such as the creation of a technical design document. Might it make sense to create a technical design document for a new, complex system interface? Absolutely. Might it make sense to create a technical design document for adding a new, clearly-defined field to an existing report? No – this is a perfect example of documentation for documentation’s sake – a common source of waste that both Agile and Lean methodologies eschew.

Most of the things that I’ve seen work are borrowed from established methodologies such as RUP, Scrum, XP, Lean and Kanban. I have no interest in an ivory tower methodology that somehow purports to be the new, unique way to develop software, period. I make every effort to include those portions of existing methodologies that I've seen work and leave behind those that I have seen fail. Beyond the contributions of formal methodologies, patterns can and should also be infused with tribal wisdom and collective experience – things that just work, regardless of whether you’ll ever read them in a book. By forming a library of patterns we can create the ability to define methodologies dynamically based on the class of project at hand or even based on the needs of individual projects themselves. This is a simple but powerful idea.

I plan to wrestle with this theme (among others) in future blog entries. And like all good ideas, this one came from someone else – it is well-articulated in SDLC 3.0, a fabulous book by Mark Kennaley, which I highly recommend.