SOA sounds hard and API sounds easy

​Before cloud computing became the hottest topic in IT recently, the big buzzword was SOA (Service-Oriented Architecture).  The business benefits of SOA can be astounding when a company actually implements a true loosely coupled architecture.  SOA enables IT to improve speed to market, reusability, flexibility, adaptability, maintainability, etc.  which helps the business drive revenue, reduce costs, on board clients faster, improve customer experience, and much more.  Unfortunately, a small percentage of those who have tried implementing SOA have pulled it off.  Why is that?

First of all, SOA is an architectural approach, not something you can buy or copy code off the internet from.  SOA is something  you do.  SOA requires change.  Change in how we architect, change in how we organize, change in how we govern, and change in how we view our business.  Why does SOA fail so much?  Because it requires change.  Change is not a technology thing.  Change is a people and process thing.  So many IT shops forgot about the later and only address the technology part.  After investing tons of time and money many IT shops simply built a ton of web services without reaping the promised benefits and without getting business buy in.

Enter the API.  An API (Application Programming Interface) is a loosely coupled language independent service that is used to expose one or many business processes to an internal or external service consumer.  In essence, it is fundamentally an implementation of abstracted business services which is the vision of SOA, at least from a technology standpoint.

Selling SOA to management is always a challenge because SOA sounds, looks, and feels like an IT thing.  Thanks to the explosion of social media applications like Facebook, Twitter, LinkedIn and others, the business understands and wants APIs.  In fact in some companies, the API is the product (i.e.​ Google Maps, Bing Search, Facebook Like, etc.).  Now IT shops are forging ahead building APIs at a rapid rate.  Sounds easy doesn’t it?

The challenge is, whether you say you are doing SOA or building APIs, it is still all about a combination of technology, people, and process.  The same issues exist for creating APIs that existed for implementing SOA.  One must still think differently about architecture and build loosely couple services that have the right level of abstraction, versioning, security, usability, etc. One must also be in alignment with the business to ensure that investments are being made in the right APIs for the right reasons and with the right requirements.  And finally, one must govern APIs differently than we have governed software from days gone past.

 

Summary

The good news is that with the surge in acceptance of social media and cloud computing, the business understands, wants, and needs APIs.  The bad news is, this doesn’t make implementing change in the organization any easier from a technology and process perspective.  It helps a little from the people perspective because more people are bought in to the concept but that doesn’t necessarily mean they will change their ways.  At the end of the day, it all boils down to good architecture.  Know your business architecture, know your business goals and desired outcomes, build the right level of abstraction, be the change that you want within the organization, and govern closely.  Call it SOA or call it API development.  Just don’t skip the steps.

Share

Myth Busting, Swiftboating, and outright lies: A day of cloud nonsense

Several months ago a CIO from a company I was meeting with claimed his ordinary data center was a private cloud.  Nonsense!  I vented with this post.  Today there are two good myth busting posts circulating (here and here).  One of the myths in Dave’s article states that putting data in the cloud could get you put in jail.  That is pure swiftboating!  But what is really getting annoying to me is watching companies and developers call their web apps cloud apps.  It seems that many people are now padding their resumes with the term cloud computing because they have written an app that runs in a browser.  To make matters worse, some of these apps only work in IE which doesn’t even qualify it as a good web app.

So let’s review (again) what the characteristics of cloud computing are (my apologies to most of my readers who are sick of reviewing these definitions):

  1. Broad Network Access - Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs) as well as other traditional or cloud-based software services.
  2. Rapid Elasticity - Capabilities can be rapidly and elastically provisioned — in some cases automatically — to quickly scale out; and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.
  3. Measured Service - Cloud systems automatically control and optimize resource usage by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, or active user accounts). Resource usage can be monitored, controlled, and reported — providing transparency for both the provider and consumer of the service
  4. On Demand Self Service - A consumer can unilaterally provision computing capabilities such as server time and network storage as needed automatically, without requiring human interaction with a service provider.
  5. Resource Pooling - The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a degree of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources, but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter)

If an application does not provide or inherit these characteristics from its service model then it is not a cloud app.  If you know someone who is spouting outright lies about their apps, have them repeat after me:

 

 

Share

Cloud Computing: Sys Admin & Security professionals’ nightmare or dream?

Yesterday I asked the question Is the Cloud driving better software?  I just can’t keep these thoughts out of my head.  What I am about to describe is what I have seen in the places that I have worked (your miles may very).  I have worked mostly in manufacturing and retail environments where security was never the top priority although I could argue that it should have been up there.

Source: (Thread of Truth)

All these years that I have been building software I have never seen so much scrutiny on security as I am seeing now.  In my early days (the 1980s) on the mainframe, there was a small team of admins and maybe one security person who controlled the world.  No developer could get a lick of code in production without going through this team.  Often this team put heavy constraints on the developers’ design regardless of what the developers’ needs where.  It was their mainframe and they where in control.  The Internet did not exist and everything ran on a private network for internal users only.  Applications were secure by default but the developers were very limited to what they could build and deploy.  Life was great for the admin and the security person (if one existed).

Then came Windows and Unix and client-server applications started sprouting up all over the place (1990s).  Admins and security personnel quickly lost the power and control of the enterprise and developers ran wild.  Gaping security holes appeared everywhere as applications sprouted up in silos in many different technology stacks with not enough application security.  Companies hired developers in droves but often the admin and security teams remained at the same size.  The end result was that infrastructure and security teams had to prioritize what projects and silos to focus on because they could not cover everything.  Life was getting better for developers.  Many of their design constraints were lifted because the control shifted from a contained central group to a distributed group of different and less powerful governing bodies.  Life for the SysAdmins and security personnel was changing. Suddenly managing the infrastructure and controlling security was much more challenging and becoming less standard.  Patching became the burden of life for admins.

Then the Internet boom took off (mid 1990s to 2000s) and all hell broke loose.  Suddenly developers were in high demand and were cranking out code like crazy that was exposed over the Internet to expecting customers and expecting cyber criminals.  The number of developers far outweighed the number of admins and security personnel who could contain them.  Gaping holes in security sprung up everywhere.  Default user IDs and passwords were exposed to the Internet.  The new breed of drag and drop rapid developers who never lived in a controlled and contained world of the mainframe or the early client-server days were unleashed with a limited amount of exposure and knowledge to the required security requirements that should have been implemented.  Web developers during the Internet bubble were hired in droves and paraded around like rock stars as admins and security teams ran around behind them like the cleaning crew after a frat party.  Senior management often could only see the justification in hiring more developers and not infrastructure and security professionals. Admins and security became background noise as bright shiny apps took to the spotlight.

Enter the cloud era (2008-current).  The benefits of cloud computing are a game changer.  Every executive wants to be in the cloud.  But many are afraid of putting customer data outside of their firewall (as if it was safe inside their under manned data center).  Suddenly, managing infrastructure and security is a hot topic again.  So my question for admins and security professionals is….is this your worst nightmare or is this a dream come true?  Those who view this as a nightmare will fade away like a Y2K Cobol programmer while those who seize the opportunity can live the “rock star” life like the mid 1990s web developers (can you say DevOps?).  Let’s discuss this.

As I mentioned in my article yesterday, the RFPs that I am responding to today are loaded with pages of questions concerning IT controls, security, compliance, data ownership, etc.  I am processing digital coupons and getting the scrutiny that one would expect in the health care industry, the banking industry, and the stock trading industry.  I personally think this is a great thing for two reasons.  First, I know my competition is not addressing security and controls to the level we are but second, as a practitioner, I am happy that companies are prioritizing application security and controls at the level that has always been merited.

Getting back to my question and reason for the article….this is the golden era for system administration and security.  Not since the early mainframe days has there been so much focus on standardization, automation, and IT controls.  The cloud is forcing people to go back to the basics and design for the things that we have taken for granted for so long.  So is moving to the cloud a nightmare for these folks or is it a dream come true where their jobs are finally back in the forefront of management’s mind?  Is the glass half full or half empty?  The optimists will seize the moment and grow, the pessimists will fade away and become irrelevant.  What will you do?

Share

Is the Cloud driving better software?

As companies shift to delivering software as a service (SaaS) or platforms (PaaS) the customer demand for increased security, closely measured SLAs, complete disaster recovery plans, and 24×7 availability is driving more rigid technical requirements.  You can thank the cloud for this.  Traditionally software companies built packaged software that only required a fraction of the technical requirements that today’s cloud services require.  Now much of the software being built is being sold to customers who are used to using Internet enabled software that is always up, always scales, is easy to use, and connects to everything.

This shift is caused by two major factors.  First, the always on/always scale nature of SaaS/PaaS solutions require deeper investments in architecture to meet customer expectations.  Second, the cloud is “new” and “scary” and executives and IT staffers are requiring much more from their vendors than ever before.  In fact, some IT shops are against the cloud but are forced by management to select a cloud solution because management refuses to build and support things that are not core to the business.  These IT shops require so much in depth technological requirements in their RFPs that it is often excessive.  In fact, often they require security at levels far beyond what they have ever delivered to their own business (I have witnessed this first hand).

All of the above mentioned factors are driving more robust solutions in the market place.  Let me count the ways:

1)      Security awareness is at an all-time high now that software can’t hide behind our corporate firewalls anymore.  Application security is finally as important as perimeter security.

2)      Cloud based software is expected to seamlessly connect to other cloud services, hence driving acceptance of standards

3)      Cloud based solutions are now expected to be offered as a collection of independent services rather than large monolithic software packages thus driving API best practices (SOA is a dirty word these days)

4)      Cloud services need to be always up/always scale thus driving deeper investments in architecture to deliver higher degrees of reliability, performance, resiliency, scalability, etc.

5)      Cloud based services often require vendors to publish SLAs and be very transparent about it.  This is driving investments in DevOps, monitoring tools, automation, and much more.

Building software for the cloud is a big shift from selling packaged software on a CD.  Much of the above mentioned technical requirements did not require as much engineering for packaged software because the software was usually managed by on-premise resources over an internal network that was totally under the control of the customer.  The software rarely changed (occasional patches), gave the mirage of being secure (hiding behind the corporate firewall), could be proprietary (did not need to connect to everyone and everything on the web), and did not need to auto scale (sys admins managed server and database capacity).  Building good cloud based software requires that we build more robust software than ever before.

Summary

The rapid adoption of cloud services is driving up the demand for more robust technical requirements.  Does that mean we are building better software?  Not really.  If we build crap in our data center than we will likely build “iCrap” in the cloud.  However, successful cloud companies will build better software because their customers will expect it and demand it. It is much easier to switch out an API than switch out a packaged solution so the days of mediocre solutions are numbered.  What I predict we will see is excellent cloud services being deployed by newer companies building complete solutions from the ground up in the cloud.  These cloud services will be always up and always scale, will be transparent about their SLAs, will connect to everything through standard use of RESTful APIs and security protocols, and will work seamlessly on any device.  Many companies whose core competencies are not building cloud based services will try to port legacy code and/or try to force their legacy processes, mindsets, vendor stacks, and worst practices to the cloud and fail miserably.  What will unfold over time is that the industry will get better at building very robust and reliable cloud based services and companies will shift to integrating with cloud services and refrain from building or buying solutions.  The companies that stay the course and shy away from the new world of cloud APIs will be left in a trail of dust.

Share

Why I did not attend the Big Data conference

There was a very well attended Big Data conference in New York a few weeks back.  I contemplated going but chose not to and here is why.  There is a highly repeated pattern for conferences when the topic of the conference is “new” and emerging.  It goes like this:

I have been through this already with BPM, SOA, and Cloud.  I remember attending a Gartner BPM conference in the very early 2000s and the vendor pavilion must have had 50+ vendors selling silver bullets.  The following year there was about 20 left.  When I attended the early SOA conferences I was amazed and appalled that many of the vendor tools that have been in existence forever were now SOA  solutions (can you say Rational?).  To make matters worse, every speaker had a different definition of what SOA was and I felt like I was at a game show waiting for “Will the real SOA please stand up” to be announced while vendors bobbed up and down until the one true SOA stood up.

When I attended the early Cloud conferences, I was already a year into architecting our own PaaS built on AWS so I had a very good understanding of what cloud computing was.  I remember sitting in the speaker room with David Linthicum as we shook our heads at the marketing drivel and FUD being spewed from stage to stage.  I wrote about it here.

I also know a thing or two about “Big Data” having worked on significant projects dealing with many Terabytes on a proprietary NoSQL database (before the term was coined) distributed across a multi-node Beowulf cluster.  This was poor man’s NoSQL/Big Data (actually it was rich man’s because we did it without spending millions of bucks).   Although this initiative did not leverage today’s favorite tools for processing “BigData”, it is the same concept that the vendors are pushing today.

When confronted with the choice to make the journey to the Big Data conference I relied on my past experiences from attending conferences touting “new” technologies.  Should I go to the conference and sit through a lot of theory, FUD, half-truths, and marketing baloney with the hope of hearing one or two real life experts share something I can learn from?  My decision was no.  Let the circus go on and wait a while until Big Data becomes more defined, more real, and more implemented.  In the mean time, I’ll continue to find real learnings on the topic through my network of technologists on Facebook and Twitter.

Share

Free API Design eBook released from Apigee

The team at Apigee release a free eBook on API design recently.  As I have written in the past, the world we live in today is all about integration thus the API is at the core of the success or failure of much of today’s  modern software.  This eBook is a must read.  For those who don’t have the time I’ll hit some of the highlights here:

These bullets are just a subset of the great information in the eBook.  The bottom line is that API developers must architect their APIs to be simple, intuitive, and quick to implement.  The easier the APIs are to understand and use the more likely developers will use them.

Share

In the cloud,one service model does not fit all

I wrote a post called which cloud service model  is right for you a few weeks back.  On Monday, 12/12 at our Cloud Camp in Winston-Salem, I will be giving this presentation on cloud service models.

As I mentioned in my previous article, my company uses a combination of all three models. We have a high speed transaction processor and a high volume consumer facing web site on AWS, we are building a few work flow style applications on Force.com, and we leverage many SaaS solutions for our non-mission critical and non-core competency tools.

There is still time to register for the free Cloud Camp if you are in the area on Monday the 12th.

Share

Admins and the cloud: Doomsday or Center Stage?

I will start with the disclaimer that I grew up in the app dev side of the house and was thrust into the DevOps side of the house as a CTO of a startup with limited resources.  I know that there are pockets of on-premise admins that absolutely despise and even fear the cloud. But is the sky falling for professionals who have spent 20+ years building and patching servers and managing complex networking architectures?  Only if they perceive that it is.

From Enterprise Initiatives

Source: http://krishna.org/2012-the-end-of-the-world/

 

The cloud has thrust the behind the scenes superstars onto center stage because without the expertise of these veterans, the cloud delivered services and applications can quickly become a major fail story.  What I find even more fascinating is that many ancient and expensive admin tools are quickly falling out of favor for new open source or SaaS solutions that have been built in recent years and don’t carry all of the baggage of old commercial software.  There has been a tremendous amount of innovation in the cloud management space and more is coming!

So let me get this straight.  If I am a DevOps kind of guy and someone gives me an opportunity to implement a new infrastructure management strategy using some of the hottest and high in demand technologies in the marketplace while learning new tools and applying my years of experience, I’m thinking this is a great thing!  Where do I sign?  Of course, for those who hate change, the cloud sucks!

The same applies to security professionals.  No longer is security, especially application security, an after thought.  Security is the CIOs number one concern with the cloud.  This puts the security professionals on center stage as well.  I was at the Cloud Security Alliance conference a few weeks back surrounded by many scary looking security folks, you know that look!  The message I heard over and over was “don’t fight it, embrace it” and “we must help secure the cloud”.  This should be the same mantra for the DevOps folks (they have that look too).  They should be saying “don’t fight it, embrace it” and “we must help manage the cloud”.

So is it doom and gloom or does the cloud thrust the back office guys into the spot light where they have always belonged, right along side the developers?  Thoughts?

Share

Cloud and on-premise cost comparisons: Apples and Oranges

I have encountered a lot of discussion lately trying to evaluate the costs of both an existing IaaS delivered solution and a proposed PaaS delivered solution to the equivalent cost of building or hosting the same on an existing data center.  In every discussion it appears that the cloud is simply more expensive.  The problem is we are comparing apples to oranges.  What we really need to do is a side by side TCO (total cost of ownership) analysis.

From Enterprise Initiatives

 

Here are the issues I run into when non-cloud folks analyze the costs of on-premise solutions.

 

In a nut shell they are simply comparing a one time cost of a server to an hourly cost of a virtual machine that has all of the above mentioned costs baked in.  So when one looks at a per user license of a PaaS or SaaS tool, they need to realize that they are getting way more than a license that one is used to getting for commercial software.  They are getting all of the benefits of not having to build the infrastructure, provide elasticity, build highly redundant and fault tolerant applications, etc.  With a traditional commercial software product, the license (plus 20% annual maintenance) gives you permission to use the software.  It does not include the costs of buying redundant servers, managing and patching those servers and software stacks, providing for disaster recovery, etc.

I also hear a lot about how restrictive the platforms are and all of the things that you can’t do on it.  What about all of the things you can do on it?  If SalesForce.com is built on Force.com and it is used by most of the fortune 100 companies and provides the proper uptime, reliability, autoscaling, security, SLAs, etc., then why is it not a good enough solution for your enterprise?  Think about that.  Don’t get me wrong, the cloud is not the answer for everything and PaaS has its limitations for applications with extreme processing requirements (IaaS is good for that), but building things yourself on-premise is not cost-effective and one could argue it is not even responsible anymore if the requirements don’t merit it.

And finally the last argument which drives me nuts is this:

” The cloud does not have robust enough development tools, testing tools, debugger, etc.”  Neither does the open source stack I use which costs substantially less than commercial software and simply powers most internet apps.  Yes the cloud is not as mature as a .Net on-premise computing environment.  But is the comparison really about developer preference or is about business drivers?  is the most mature and familiar tools always the best tools?  Cobol is pretty mature.  I don’t see people running to that anymore.  To borrow a phrase from the ESPN Football Countdown, “Come on, Man!”

Share

SOA anti-patterns revisited

As I mentioned in a blog post earlier, I am engaging in yet another enterprise wide SOA initiative and would be blogging internally sharing my lessons learned. In the name of reuse (pun intended) I figured I would also post here on my blog. My apologies to all of those fatigued from years of SOA discussions. Here it goes….

 

SOA Anti-Patterns

This is my first blog post in a series of many about SOA.  As some of you may know, this is my third go around with Service Orientation.  I have learned a lot of things by failing first and would like to save you all from having to learn the hard way too.  That is not to say my previous SOA initiatives were failures.  They have their share of success stories.  But the road to success was much harder and more costly than it needed to be.  With that said, let me share an ancient classic blog post from SOA expert, Steve Jones, written way back in 2006.

SOA Anti-Patterns

There are a few anti-patterns that I want to focus on because I feel they are relevant for our situation.

Anti-pattern: The technology altar

We have had passionate debates on the need to talk to the business about SOA or tie our SOA initiatives to key business objectives.  Doing SOA in an IT vacuum results in what I call doing “IT for the sake of IT”.  It is important to remember that the reason IT exists is to support and enable the business, period.  The business, either directly or indirectly, pays for everything IT does.  It is critical that our SOA initiative is attached at the hip with some key business benefits.  We all know the IT benefits of SOA: reuse, better integration, reduce maintenance, etc.  But how do these IT centric benefits get our product owners excited?  They don’t.  What product owners want to hear is increased sales, faster customer on-boarding, improved customer satisfaction, etc.  SOA can provide all of that if we have a well-defined roadmap of projects that drive business value through the use of service-orientation.  Sure we don’t need to ask the business if we should do SOA or not, but how will we fund SOA (the tools, the overhead, the governance, the design time, the people, etc.) if our product owners don’t understand what they get in return?

My recommendation:  Make sure your SOA initiative is aligned with key business drivers that are tangible and measurable.

Anti-pattern: A million services in a row

This pattern is a killer and either leads to slow and unusable applications and/or incremental hardware purchases to make up for poor design.  What happens here is developers get passionate about services and decide that everything must be deployed as a service.  Now I am a big fan of services, but services carry some baggage.  First, they take more time to design properly.  Second, they can create a hit on the overall performance.  Third, they create more demand on stricter governance and oversight.  In other words, services are expensive and should only be considered when the consumption of the services merits it.  For example, if a service is likely to only be used in one place by one application, why is it a service?  Why did we spend the time to build it and the ongoing money to manage and maintain it?  Also, if everything is built as a service, there is probably a lot of duplication occurring across development teams.  An analogy I like to use is the comedy skit where two guys line up for a fight.  The one guy starts performing all kinds of fancy martial arts moves by spinning, kicking, jumping, and hollering.  When he finally attempts to strike the opponent, the opponent simply pulls out a gun and shoots him.  The moral of the story is make sure you bring a gun to a gun fight!  This anti-pattern, closely relates to the anti-pattern: Nobody’s home.

My recommendation:  Do an analysis and create a list of candidate services.  Evaluate the value of each service independently and only create a service if it is likely to be highly reused and it provides value to the overall business goals.  Make sure there is a reward for the effort!

 

…and here are a few patterns that I came up with

Anti-pattern: Testing is as testing does (apologies to Forrest Gump)

Many companies ignore the fact that moving to a service oriented architecture not only changes the way we design and develop, but it also changes the way we should approach testing.  Duh!  So I have witnessed companies pour money into tools and developer training only to drop the new whiz bang code into a QA environment and are surprised when the tester has no idea why it doesn’t work, or worse yet, if it works or not.  In fact, testing of web services should be performed as a stand-alone process before the service is allowed to be consumed by any service consumer.  In addition, these services should be versionable and backwards compatible, which also needs to be validated by QA.

My recommendation:  Include the testing organization in your plans from the start.  Train them and shift towards a more XP model where the developer and the tester work hand and hand on services.  Build test harnesses and other tools to help validate that the services work before they are published for consumption.

 

Anti-pattern: Death by governance

Another common SOA killer is a group of ivory tower architects or development purists start with very rigid and robust governance plans.  They apply too much governance too soon.  The first problem is that they spend the first several months and sometimes years delivering no business value whatsoever because the time is spent arguing semantics.  When they finally do settle on the rules of the game, they now need all kinds of people and tools in place to manage an empty repository.  Governance is something that should have its own roadmap and should start small and grow at the speed of the maturity of the SOA initiative.  In other words, the amount of governance required to manage the first 10 services is minimum compared to the amount of governance required to manage 100 services.  Also, the demands on run-time governance (service monitoring, versioning, SLA management, policy enforcement, etc.) are not a line that runs parallel with the number of services built.  It is more like a hockey stick where one can get by with little to no run-time governance early on but before you know it the whole house of cards comes crumbling down as service consumption takes off.

My recommendation:  Carefully build a parallel roadmap for your governance model.  Practice “just enough governance” and evolve your processes and people over time.

…and finally, my favorite and the one that has left me scarred for life….

 

Anti-pattern: Get off my lawn!

The biggest SOA killer of them all is failure to anticipate and effectively manage organizational change.  Although we can argue that SOA is nothing new, we can’t argue that embracing SOA creates changes to the way people across the organization perform their jobs.  Anytime people are asked to change, they need to know why, what is in it for them, why it will make things better, what it will look like when it is done, and many more questions.  Often it requires deep cultural changes, with the most common being eliminating organizational and application silos and thinking more holistically.  There will always be a movement against change.  An organization must quell that movement by constantly communicating the vision and value of the initiative.  Some people will never come around and buy into the vision  If these people are influential, they need to be coached or removed.  Enterprise wide initiatives are hard enough when everyone is all in.  If influential people are revolting, they create issues which lead to increased costs, delays, and sometimes downright failure.

My recommendation: Get senior level buy in.  Create an organizational change management and communication plan and have a separate roadmap for this that is funded accordingly.  The best money spent is money spent on driving change.  Without it, change will be hard to come by.

 

 

Share