Agile Software Development contributes to flow experiences

I've been practicing agile software development since the early XP days. When the agile manifesto came out it seemed to echo many of the things that I found good about agile software development. For our shop we needed agile software development so we could eliminate waste in the form of unneeded features which contributed to code bloat. To eliminate this waste required the participation of our customers. Customers would meet with us daily in a variety of ways, email, face to face, or in instant messenger. They would see the progress we made on a particular feature and say "good enough" or propose corrections. All in all it was a very efficient process. Likewise, as a manager I met with the business weekly to ensure that what we were working on was aligned with their objectives. In an ideal world, their objectives would line up with the company's strategic plans. This was a perfect synergy of strategy, tactical objectives to achieve the strategic goals, and software development addressing the objectives in two week releases of new features. I often thought about the unexpected by product of this whole process; happy customers and more importantly happy staff. It seemed that no matter how daunting a task I gave my team, they just smiled and got down to work on it. Truly, this was a software development manager's dream. Little did I realize that the happiness was a byproduct of 'flow' and that flow was a direct result of agile software developments' other tenets, namely self-directed and self-organizing teams.

Consider the following principles as described on the Agile Manifesto website:

• The best architectures, requirements, and designs
  emerge from self-organizing teams.
  
• At regular intervals, the team reflects on how
  to become more effective, then tunes and adjusts
  its behavior accordingly.
  
• Build projects around motivated individuals.
  Give them the environment and support they need,
  and trust them to get the job done
  

As a software development manager I was encouraged to turn over command decisions to my team so that they could come up with the architectures, requirements, and designs. My job was to accurately relay the problem and manage any impediments to progress. Our team would get together monthly and review what worked and didn't work and adjust accordingly. My job changed from getting hands on with the code, to supporting my staff, hiring the best and brightest to join our team, and understanding the needs of the business as deeply as possible.

Our team evolved into a very dynamic, highly cohesive, unit that would work together, eat together, and celebrate victories together. From the first design principle, people well versed in architecture started to shine and teach others about better ways to develop software in accordance with design principles and patterns. We moved from good ASP.NET developers to really good object oriented developers. This transition helped us bridge the religious divides of Java and .NET as we quickly started to attend tech talks and pattern workshops with the Java guys (they called us the 'sharpies' (for C Sharp (C#)) and we called them the 'dullies'). How was all this openness, honesty, and good work ethic achieved? Was it the agile software development methodologies we were following? No, it was the fact that in spite of all the hard work, we were all very happy. We took IMPOSSIBLE requests and turned them around in weeks, not months. We were by all measure one of the most productive groups in all of IT at the time. So where did all of this happiness come from? In a word, FLOW.

So, what is 'FLOW'? Flow is a concept described by Mihaly Csikzentmihalyi. I could spend this entire post describing it. But I would refer you to two really good articles on the subject, Frank Heckmans' article on 'Designing organizations for flow experiences' in the Journal for Quality and Participation, and an excellent post by blogger Steve Pavlina on the 7 Rules for getting into flow. The simple description of flow is the experience of happiness, energy, and creativity when someone is perched between impossible goals and insipid, easily achieved goals. In that fine line of tension, energy manifests itself, creativity is forced into play to achieve the 'nearly' impossible, and people just feel in the 'zone', invincible, and most importantly, HAPPY. There's a growing body of research that supports this psychologically, sociologically, and even anthropologically. It seems the state of flow is a very human experience from our antiquity. Flow was something that developed in the early nomadic tribal groups, wherein members were all called upon to participate, lead, follow, and play different roles.

My agile software development teams seem happier than other non-agile teams. I believe it's because agile software development in whichever methodology you choose, fosters flow. Its practices contribute to and create opportunity for flow experiences. Agile software development does this by way of its aforementioned principles, self directed and self organizing teams. These mirror the research done by Emery on the design principle of work organization that leads to flow; self-managing, adaptive group structures. Like the many links one sees to the Fibonacci sequences in nature and the universe, self-organization has roots in autopoiesis, complexity theory, physics, and biology. It would seem that self-organization is the natural order of things and when human beings at work in any area follow it, they are happy.

Emery's research suggested six requirements for human productivity. I will translate these to aspects of agile software development and how managers should structure the organization around that team for maximum productivity and employee satisfaction.

1. Autonomy – Self organizing teams are self-directed. You may suggest problems for them to solve, challenge them on the accuracy or completeness of their solution, but you can't employ traditional top-down management with them. You have to give them support and guidance so that they get needed direction enough not to feel lost or bored, but not so much that they feel like they have no ability to influence their success in the organization.
   
2. Foster a learning environment – This is something Google does exceptionally well in theory, they encourage staff to learn new things as long as they are loosely related to the company's overall aims. Agile teams constantly challenge one another to do more, work smarter, and learn new ways of being more productive. This makes people happier as a result.
   
3. Variety – Cross functional teams are advocated in agile software development. Cross training and mentoring are hallmarks are these teams. This leads to variety which helps people stave off boredom. Beyond that, as managers we should constantly be exploring new technologies and encouraging our staff to do the same. It's not technology for technology's sake, it's applied when applicable and at the very least the lessons learned can be applied to the employee's current assignment.
   
4. Mutual support and respect – Loyalty to the team, mentoring, and a healthy respect for each other's various proficiencies are keys to success in software development. Teams who have mastered this are able to cover for one another in absences, pick up and help one another without direction from management, and can self-organize along respective strengths better. Again, the end result is a feeling of belonging, a feeling not of us against them, but rather a nearly familial pride in each other. Agile software teams achieve this by virtue of cross-training each other, mentoring each other, and working as a team on their goals.
   
5. Meaningfulness – This one is harder to quantify for some managers. The suggestion here is that people like to think that their work makes a difference, contributes to society, and helps their employer's bottom line. How does agile software development help to achieve this? By releasing software in regular, incremental intervals. Agile teams release software every 2-4 weeks on average. Projects which take 60 days or a little more are rare. Project lasting 90 days are almost unheard of among agile teams. This means that software developers see their efforts, their code, and their creation, go into use. They get regular, almost daily feedback from their customers about how well the software works. Agile developers make a difference in their customers' lives and as such they derive meaning from their work.
   
6. Upward mobility – Some software developers feel stifled in their positions. They feel like they are spinning their wheels. Their software never sees the light of day and they will never get promoted or receive recognition. Agile software developers on the other hand know they make a difference. They can feel themselves growing as they get mentored, as they pair program, as they evolve newer and more efficient ways of writing software. Just the fact that they are on an agile team or can add experience with agile software development methodologies to their resumes is a huge plus. This in combination with the ever growing list of achievements added to their resume makes them more marketable. The really wise software development manager will promote from within, recognizing his or her employees' growth thus extending their tenure at the company. Likewise, be sure to backfill these promotional vacancies with junior level new hires that show an aptitude for learning and a 'go-getter' mentality. Lastly, as a good software development manager, recognize that at the apex of your senior staff's development, they will leave you for larger, more lucrative projects with national or global companies. Rather than trying to woo them back with unreasonably high counter offers or worse, veiled threats of sabotage or retaliation, celebrate their success within your team. This will build your personal network of former developers (who can buy you sushi someday) and will send a signal to your existing staff that their jobs are not dead ends, that they too can expect a celebratory farewell lunch someday.
   

So what will happen if you follow my advice? Will you get rich? No. You will not be personally famous. But, if you do decide to adopt agile software development (agile with a small a) then you will most likely have happier, more productive employees. It will help people that work for you get into the 'flow' and that experience by all accounts is meaningful, memorable, and makes a difference in people's lives. And if you're lucky, you may just end up on a list of some of the best places to work in IT, over and over again.

Ubuntu in the office

So yes, I've been running Ubuntu (6.10 and now 7.04) in the office since October. I've definitely had my challenges but now I rarely need to cross over to Windows. The one thing that vexes me is a good substitute for Visio. I've got UML tools and Thinkature
but thats about it. So how do I get by with Ubuntu in a largely Windows corporate world? Here's how:

1. Buy in. My management is completely cool with my running Linux instead of Windows. I exist in open source goodness by the grace of their positional authorities. If you dont have buy in, you're asking for trouble, run Ubuntu as VMware or on a thumb drive.
2. Email. Well Outlook seemed to be an insurmountable challenge. I have Outlook Web Access which is pretty damn swell in Exchange 2003. But I've forced myself to know Evolution and I've been pleasantly surprised with how well it performs on Feisty Fawn. Its everything I need in an email client but it still has it quirks. It occasionally hangs, or crashes without saving my appts to my calendar, etc. But overall, no issues.
3. Development. Well this is afterall what I am supposed to do. I have been forced to use MonoDevelop in lieu of VS.NET 2005. I like MonoDevelop, but I am limited to doing just source code and very little complex stuff. I can check it in through KDESVN *I still say Tortoise is WAY better. And even though MonoDevelop is based on my all time favorite OSS IDE for .NET (SharpDevelop (I LOVE YOU BABY!)) its no where near as good as VS.NET 2005 or SharpDevelop yet. I also use Eclipe with a variety of plugins for Ruby, BPEL, UML, and many more. I really really like what they've done with Eclipse. Now if they could just get the refactoring thats built into NetBeans or a plug-in like ReSharper (the BEST Visual Studio plug in EVER) I would never leave. AND it runs SOOO much faster on my Ubuntu than Windows ever did for me.
4. Web. Oddly enough there are still times when I wish for IE back. I know its odd to say but for some Microsoft only solutions, its king. I was pleasantly surprise to see Sharepoint 2007 work well with FireFox 2.0. Nicely done Microsoft. And yes, I did install IE for Linux *v6 but sill dont like it as much as the real thing.

Conclusions? Ubuntu, OpenOffice, a SLEW of development tools....Ubuntu is great. Challenges? Enterprise buy in and integration challenges, no corp VPN, no corporate virus protection, no Outlook. If I have a problem with it, I own it. Other challenges, HARDWARE. Driver issues plague open source systems. Try to get full compatiblity with NVidia or worse yet ATI with Ubuntu. Got a Web CAM? Try to use it with GAIM or aMSN....its not that easy. Ubuntu is the best Linux yet for the desktop. For open source shops or non-Microsoft developers, this is a real consideration.

Lessig's Presentation

Aside from the brilliance of his presentation style, Lessig has an important message. His message was that an internet 9/11 is coming, which will result in an internet version of the Patriot act. This work is already under way with the Identity gang and Kim Cameron at Microsoft. The identity layer, an identity metasystem will enable traceability and regulability. Lessig's call for action was to have we as technologists involved in framing and shaping the discussion in order to preserve the generative, free, Internet we enjoy currently. He offered no clear way for us to get involved short of getting off the PC and getting into the political arena to help counter the perception of the Internet as a series of 'tubes'.

I also got to ask him about the CC license and the Microsoft Zune. His reply was that DRM much like the Patriot Act was a poorly crafted, poorly implemented technology. He suggested that the conversation with Microsoft and Apple (even Apple is guilty of catering to the DRM Nazi's) has been started and that its an important one for the digital rights world.

Gartner Identity Management Conference Summary

WOW. I am really proud. Apollo is ahead of the curve in SOOO many ways. Now mind you we're not at the apex of Gartner's Maturity Model which is Policy Based (and yes that is our new target) BUT...

We're at the Virtualized stage, our objectives now are increasing business efficiency to reduce costs in labor intensive or time intensive business activities.

A whopping 73% of Gartner customers DON'T do automated user provisioning. Likewise, all the products we looked at have CRUDE interfaces, ill defined interactions, and are still half baked. Even Oracle, the best identity and access management suite on the market today is only a rebranded amalgam of their most recent acquisitions. It introduces yet another workflow technology, it has gaps in what it can and can't do, namely business roles and role governance. For that it recommends we leverage someone like Bridgestream, which has yet another web interface for business users to use in requesting business roles (apparently only IT users use Oracle Identity Manager to request IT only roles) and yet another workflow engine for our support teams to learn. Add to that the spartan and un-intuitive interfaces on both products, and we look like rock stars.

We are in fact rock stars, what we've done and in the time that we've done it is nothing short of miraculous. We're in the top 27% of Gartner customers for user provisioning and I'm quite sure we're even higher considering the periodic audits we've been doing for a full year this month. At the user round table I attended Thursday, I was ahead of all but one customer and even then we had features and maturity that they were only now starting to consider.

So, considering all of this, where do we go next? Well there are still significant gaps in our offerings, and most certainly, opportunities for growth. Here's the short list of things we're missing or needing to improve upon:

UI: what we've got in the CAP UI is extensible, robust in comparison, far easier to user and more elegant than anything we saw at the conference. But being the rock stars we are we cant settle for success. Let's take it to the next level, let's simplify the UI and make it clean. We'll engage the HCI team in order to get this done. Think of the good Web 2.0 designs we've seen like Google, Skype, Delicious, and you'll have an idea of what it is that we want to do.

On-boarding: We need to simplify the on-boarding process for all new users. CAP can still be the place we go for on-boarding contractors. But when we hire staff or convert contractors to staff we HAVE to vastly simplify and streamline that process. Users shouldnt have to go from HR to CAP to put in access requests. Likewise, we should be able to identify people as existing identities when we provision them so we dont end up with duplicates. Every vendor has this, so should we.

Default access levels: When we on-board people we should automatically grant them a default level of access based on a role. This access would include network access, email, and some combination of roles based on their job code and cost code.

Email access: Its not fully automated provisioning until we include email. This is #1. Enough said.

Role Management: We need a means of adding or removing roles within CAP, the identity management environment, and downstream in the applications. This is a larger, multi-year goal but one we should pursue nonetheless. We should include some manor of reporting all the roles and role mappings in the system as well as who has these roles and who is in violation of the conflicting role policy. Role policy shouldnt be an Excel spreadsheet. That's just plain embarassing for a rock star.

Role simplification: We need to work on reducing role proliferation and streamline what we have to be more reflective of the true business roles. We should include some definition of what exists withing EMS.

LONG TERM GOALS:

RFP for IAM Suite: We should look at what we have versus what's available in the industry. We'll need to get some scope definition in place, then we'll engage Gartner to craft a proper RFP. Once we've got that we'll send it to the major players to see who responds. I'm anticipating we'll evaluate IAM Suites from Oracle, Sun, Microsoft, and IBM, with the outside possibility of BMC. We should narrow that down to 2 vendors within 3-4 months and then do POC's with both. Based on the results of the POC, we'll select a vendor and engage Purchasing for the contract. Here's the timeline for the RFP:

December - January: Scope Definition. We'll need to work on getting the list of applications we have as an enterprise, then list what it is we have in terms of IAM support for all of them. Once we've got that we'll draft the RFP with Gartner.

February - March: Draft the RFP with Gartner. Vet it with the business. I'm anticipating 2-3 days with Gartner on site potentially.

April - May: Send out RFP and await responses.

June - August: Vendor on site meetings, demo's, and selection. I'm anticipating we'll make #1 and #2 offers by the end of August.

September - October: POC one and two.

November - December: Work with Purchasing to sign contracts. Begin to plan phased roll out.

IMPORTANT NOTE: An important option we have open to us throughout the RFP process is to pass on all vendors before or after the POC(s). The relative immaturity of the market, the relatively high prices, and the relative maturity of Apollo's IAM infrastructure by June of 2007 could suggest we pass on all vendors for 2007-2008.

The rationale behind the drive to select a single vendor or at minimum 2 partial vendors is to reduce the manpower needed to build and deliver identity and access management as well as to move away from something thats completely customized and labor intensive to own and operate. Our development resources should be able to get to a point where they are focusing on integration and delivering services to our fellow application developers in accordance with the service based model. This is very high level, very rewarding work. And then finally, when we're ready to tackle transitioning to a policy based model, our developers will very likely use a single vendors tool(s) to assist our business end users in defining and implementing their business policies in terms of identity and access management policies.

Presenting at IAM2: Next year, I want to be on stage at Gartner in Los Angeles talking about the best company with the best IAM team on the planet. I want to wow our fellow Gartner clients. I want them to base a case study on us. I want us to have to wear shades on stage that day, not because the lights are too bright, but because our FUTURE is SO bright we've got to wear shades.

Start of a new blog life

Here's my blog which is tied into Gmail and GTalk. Only fitting as I explore the netherworld of Open Source technology and programming languages