Agile Software Development contributes to flow experiences

I've been practicing agile software development since the early XP days. When the agile manifesto came out it seemed to echo many of the things that I found good about agile software development. For our shop we needed agile software development so we could eliminate waste in the form of unneeded features which contributed to code bloat. To eliminate this waste required the participation of our customers. Customers would meet with us daily in a variety of ways, email, face to face, or in instant messenger. They would see the progress we made on a particular feature and say "good enough" or propose corrections. All in all it was a very efficient process. Likewise, as a manager I met with the business weekly to ensure that what we were working on was aligned with their objectives. In an ideal world, their objectives would line up with the company's strategic plans. This was a perfect synergy of strategy, tactical objectives to achieve the strategic goals, and software development addressing the objectives in two week releases of new features. I often thought about the unexpected by product of this whole process; happy customers and more importantly happy staff. It seemed that no matter how daunting a task I gave my team, they just smiled and got down to work on it. Truly, this was a software development manager's dream. Little did I realize that the happiness was a byproduct of 'flow' and that flow was a direct result of agile software developments' other tenets, namely self-directed and self-organizing teams.

Consider the following principles as described on the Agile Manifesto website:

• The best architectures, requirements, and designs
  emerge from self-organizing teams.
  
• At regular intervals, the team reflects on how
  to become more effective, then tunes and adjusts
  its behavior accordingly.
  
• Build projects around motivated individuals.
  Give them the environment and support they need,
  and trust them to get the job done
  

As a software development manager I was encouraged to turn over command decisions to my team so that they could come up with the architectures, requirements, and designs. My job was to accurately relay the problem and manage any impediments to progress. Our team would get together monthly and review what worked and didn't work and adjust accordingly. My job changed from getting hands on with the code, to supporting my staff, hiring the best and brightest to join our team, and understanding the needs of the business as deeply as possible.

Our team evolved into a very dynamic, highly cohesive, unit that would work together, eat together, and celebrate victories together. From the first design principle, people well versed in architecture started to shine and teach others about better ways to develop software in accordance with design principles and patterns. We moved from good ASP.NET developers to really good object oriented developers. This transition helped us bridge the religious divides of Java and .NET as we quickly started to attend tech talks and pattern workshops with the Java guys (they called us the 'sharpies' (for C Sharp (C#)) and we called them the 'dullies'). How was all this openness, honesty, and good work ethic achieved? Was it the agile software development methodologies we were following? No, it was the fact that in spite of all the hard work, we were all very happy. We took IMPOSSIBLE requests and turned them around in weeks, not months. We were by all measure one of the most productive groups in all of IT at the time. So where did all of this happiness come from? In a word, FLOW.

So, what is 'FLOW'? Flow is a concept described by Mihaly Csikzentmihalyi. I could spend this entire post describing it. But I would refer you to two really good articles on the subject, Frank Heckmans' article on 'Designing organizations for flow experiences' in the Journal for Quality and Participation, and an excellent post by blogger Steve Pavlina on the 7 Rules for getting into flow. The simple description of flow is the experience of happiness, energy, and creativity when someone is perched between impossible goals and insipid, easily achieved goals. In that fine line of tension, energy manifests itself, creativity is forced into play to achieve the 'nearly' impossible, and people just feel in the 'zone', invincible, and most importantly, HAPPY. There's a growing body of research that supports this psychologically, sociologically, and even anthropologically. It seems the state of flow is a very human experience from our antiquity. Flow was something that developed in the early nomadic tribal groups, wherein members were all called upon to participate, lead, follow, and play different roles.

My agile software development teams seem happier than other non-agile teams. I believe it's because agile software development in whichever methodology you choose, fosters flow. Its practices contribute to and create opportunity for flow experiences. Agile software development does this by way of its aforementioned principles, self directed and self organizing teams. These mirror the research done by Emery on the design principle of work organization that leads to flow; self-managing, adaptive group structures. Like the many links one sees to the Fibonacci sequences in nature and the universe, self-organization has roots in autopoiesis, complexity theory, physics, and biology. It would seem that self-organization is the natural order of things and when human beings at work in any area follow it, they are happy.

Emery's research suggested six requirements for human productivity. I will translate these to aspects of agile software development and how managers should structure the organization around that team for maximum productivity and employee satisfaction.

1. Autonomy – Self organizing teams are self-directed. You may suggest problems for them to solve, challenge them on the accuracy or completeness of their solution, but you can't employ traditional top-down management with them. You have to give them support and guidance so that they get needed direction enough not to feel lost or bored, but not so much that they feel like they have no ability to influence their success in the organization.
   
2. Foster a learning environment – This is something Google does exceptionally well in theory, they encourage staff to learn new things as long as they are loosely related to the company's overall aims. Agile teams constantly challenge one another to do more, work smarter, and learn new ways of being more productive. This makes people happier as a result.
   
3. Variety – Cross functional teams are advocated in agile software development. Cross training and mentoring are hallmarks are these teams. This leads to variety which helps people stave off boredom. Beyond that, as managers we should constantly be exploring new technologies and encouraging our staff to do the same. It's not technology for technology's sake, it's applied when applicable and at the very least the lessons learned can be applied to the employee's current assignment.
   
4. Mutual support and respect – Loyalty to the team, mentoring, and a healthy respect for each other's various proficiencies are keys to success in software development. Teams who have mastered this are able to cover for one another in absences, pick up and help one another without direction from management, and can self-organize along respective strengths better. Again, the end result is a feeling of belonging, a feeling not of us against them, but rather a nearly familial pride in each other. Agile software teams achieve this by virtue of cross-training each other, mentoring each other, and working as a team on their goals.
   
5. Meaningfulness – This one is harder to quantify for some managers. The suggestion here is that people like to think that their work makes a difference, contributes to society, and helps their employer's bottom line. How does agile software development help to achieve this? By releasing software in regular, incremental intervals. Agile teams release software every 2-4 weeks on average. Projects which take 60 days or a little more are rare. Project lasting 90 days are almost unheard of among agile teams. This means that software developers see their efforts, their code, and their creation, go into use. They get regular, almost daily feedback from their customers about how well the software works. Agile developers make a difference in their customers' lives and as such they derive meaning from their work.
   
6. Upward mobility – Some software developers feel stifled in their positions. They feel like they are spinning their wheels. Their software never sees the light of day and they will never get promoted or receive recognition. Agile software developers on the other hand know they make a difference. They can feel themselves growing as they get mentored, as they pair program, as they evolve newer and more efficient ways of writing software. Just the fact that they are on an agile team or can add experience with agile software development methodologies to their resumes is a huge plus. This in combination with the ever growing list of achievements added to their resume makes them more marketable. The really wise software development manager will promote from within, recognizing his or her employees' growth thus extending their tenure at the company. Likewise, be sure to backfill these promotional vacancies with junior level new hires that show an aptitude for learning and a 'go-getter' mentality. Lastly, as a good software development manager, recognize that at the apex of your senior staff's development, they will leave you for larger, more lucrative projects with national or global companies. Rather than trying to woo them back with unreasonably high counter offers or worse, veiled threats of sabotage or retaliation, celebrate their success within your team. This will build your personal network of former developers (who can buy you sushi someday) and will send a signal to your existing staff that their jobs are not dead ends, that they too can expect a celebratory farewell lunch someday.
   

So what will happen if you follow my advice? Will you get rich? No. You will not be personally famous. But, if you do decide to adopt agile software development (agile with a small a) then you will most likely have happier, more productive employees. It will help people that work for you get into the 'flow' and that experience by all accounts is meaningful, memorable, and makes a difference in people's lives. And if you're lucky, you may just end up on a list of some of the best places to work in IT, over and over again.

Ubuntu in the office

So yes, I've been running Ubuntu (6.10 and now 7.04) in the office since October. I've definitely had my challenges but now I rarely need to cross over to Windows. The one thing that vexes me is a good substitute for Visio. I've got UML tools and Thinkature
but thats about it. So how do I get by with Ubuntu in a largely Windows corporate world? Here's how:

1. Buy in. My management is completely cool with my running Linux instead of Windows. I exist in open source goodness by the grace of their positional authorities. If you dont have buy in, you're asking for trouble, run Ubuntu as VMware or on a thumb drive.
2. Email. Well Outlook seemed to be an insurmountable challenge. I have Outlook Web Access which is pretty damn swell in Exchange 2003. But I've forced myself to know Evolution and I've been pleasantly surprised with how well it performs on Feisty Fawn. Its everything I need in an email client but it still has it quirks. It occasionally hangs, or crashes without saving my appts to my calendar, etc. But overall, no issues.
3. Development. Well this is afterall what I am supposed to do. I have been forced to use MonoDevelop in lieu of VS.NET 2005. I like MonoDevelop, but I am limited to doing just source code and very little complex stuff. I can check it in through KDESVN *I still say Tortoise is WAY better. And even though MonoDevelop is based on my all time favorite OSS IDE for .NET (SharpDevelop (I LOVE YOU BABY!)) its no where near as good as VS.NET 2005 or SharpDevelop yet. I also use Eclipe with a variety of plugins for Ruby, BPEL, UML, and many more. I really really like what they've done with Eclipse. Now if they could just get the refactoring thats built into NetBeans or a plug-in like ReSharper (the BEST Visual Studio plug in EVER) I would never leave. AND it runs SOOO much faster on my Ubuntu than Windows ever did for me.
4. Web. Oddly enough there are still times when I wish for IE back. I know its odd to say but for some Microsoft only solutions, its king. I was pleasantly surprise to see Sharepoint 2007 work well with FireFox 2.0. Nicely done Microsoft. And yes, I did install IE for Linux *v6 but sill dont like it as much as the real thing.

Conclusions? Ubuntu, OpenOffice, a SLEW of development tools....Ubuntu is great. Challenges? Enterprise buy in and integration challenges, no corp VPN, no corporate virus protection, no Outlook. If I have a problem with it, I own it. Other challenges, HARDWARE. Driver issues plague open source systems. Try to get full compatiblity with NVidia or worse yet ATI with Ubuntu. Got a Web CAM? Try to use it with GAIM or aMSN....its not that easy. Ubuntu is the best Linux yet for the desktop. For open source shops or non-Microsoft developers, this is a real consideration.

ILM Core Concepts and Architecture

Challenges: No formal relationship between the language of the enterprise and the system. We don't know what we dont know. People interpret policy. Policies are interpreted. Policies are often nebulous. Systems cant understand policy unless people who program them understand policy.

Wait, isnt this an “under the covers” look at ILM2? Don't give me marketing technobabble!!!

Communication = Contracts

Workflows = Process

Policy = ????

Assertions: Common Patterns

• Provisioning
• Group Management
• Password Management
• White Pages
• Policy Management
  

“We react to events on sets of identity with processes.”

What are the issues presuming this axiom is true

• collection of resources
• request model on collection
• sets which organize data
• blah blah blah

ILM 2 Steps

1. Resources you want to manage

• regulate your processes
• synchronize with environment
• CRUD on identity in ILM2
• core object types and schema
• you can modify types or create new ones

2. requests on resources has three phases

• authorization
• authentication
• action

3. sets organize resources
4. events trigger processes

• request events are defined as a quad
• transition events are a double
  processes formalize responses to events

5. Demo

Now Demo of the tool...again

What happened to architecture?

What if the language you create to mirror the business process isn't granular enough to reflect your needs?

how do you manage these processes? You dump them and they have to resubmit?!?! NFW!!

Clients Outlook, Sharepoint, Windows, Custom, Office 2007 (IE 6, Firefox, Windows XP SP2+)

ILM 2 will use web service standards (Windows Communication Framework, Windows Workflow Foundation)

ILM 2 will be server heavy up to 5 servers minimum – Sharepoint (WSS not MOSS), Resource Management Service, Resource Management DB, Metadirectory Service, MetaDirectory DB, **Exchange Server 2007**

Longhorn Server Standard with SQL 2005 SP2+ 64 bit ONLY

They MIGHT cluster MIIS **NO COMITTMENT and NO commitment on SQL Clustering with Support on MIIS

No event based model for MA's. No error handling for MIIS. Here's what we get maybe

“codeless” provisioning, improved performance, more adapters

The Impact of ILM

Opinion – Identity Management is a set of business processes and a supporting infrastructure for the creation, maintenance and use of digital identities.
Inherent IdM is after service based IdM? Whats Inherent IdM? Its the entire IDM Business Suite.
Card Spaces (CardSpace, OpenID, etc) is coming soon. How soon? Gerry says 18 mos. Pam Dingle says 3-4 years. I believe Pam. No offense Gerry but I'm not talking about nascent efforts, I'm talking a mature, ready for the business technology.
User Provisioning:
“Integrated set of tools for managing life cycle of user entitlements”
Components: Workflows, administrative tool integration, password management, rule based processing, auditing and logging, connectivity to AD/LDAP/etc and Identity Repository.
Leaders of IdM: CA, IBM, Oracle are tops
Microsoft and Novell make quite a few claims but they cant back it up. Workflow is the gating factor to these two vendors being in the top end. Business drivers, process improvement will happen this year. Really? no proof of that....this is a bad version of what I saw at the Gartner Conference. Ah...he's suggesting that virtualization of the IdM tools will have this effect, he's not suggesting we're re-engineering business processes yet. Any implementation experience is a multi-year engagement .
Standards
SPML v2 limited adoption, more important to Federation
BPEL Oracle and Intalio are the only two vendors, only important as Oracle is pushing it. He views this as a workflow engine for business rules, Sun may support BPEL over time. What about BPM?
Trends – Vendors still in acquisition mode. Vendors will start to work on early adopters to smooth experience for mainstream users over time. (2-3 yrs) **WAY TOO MUCH TIME on IdM deep dive. Get to the ILM bashing MAN!**
Ahh....now to ILM (a potentially disruptive technology) Microsoft has been slow – no workflow, no user interface capabilities, etc. Good metadirectory and data synchronization technology. ILM2 will be Microsoft's FIRST move into real IdM. Workflow, Web Services API (but no SPML Support) Any impact? NONE. Role management Bridgestream, BHOLD, Eurekify, Omada, Vaau. Roll your own: Courion and Voelcker, Oracle will likely acquire some of these small vendors
Enterprise Access Control Management – Logical applicationss like Approva, SAP/Virsa. Provisioning lacks awareness within ERP Stack. It checks with EACM policy engine before provisioning. Auditing tools are also big as well.
Conclusions: ILM going different direction than the rest of the UP market. ILM will be heavily partner reliant, MSFT is in this long term, don't expect them to be chased out of the market. This is a rehash of things I already knew with Gartner

Care and Feeding of your MIIS SQL Server Database

So far the best presentation of the conference *sorry Brad
Highlights:
Clear out the run history of all your jobs. This can really impact your jobs if you let it get cluttered

Spedometer - Use PerfMon
MIIS Objects
For Imports Objects Read/Sec
Objects Synchronized/Sec
For Exports Objects Exported/Sec

But wait, there's a script: http://www.ilmbestpractices.com/Articles/Speed_of_Past_Runs

as tables grow - Page splits, fragmentation, more scans than seeks, growth of log and data files.

monitor with Page Splits/Sec in perf mon how do we avoid it

scan v seek you get more scans as get more fragmented indices

read entire tables versus using an index

monitor with PerfMon

Full Scans/Sec
Auto Create/Update stats are enabled

File Growth
if auto growth is enabled
clear run history script on website

http://www.ilmbestpractices.com

DBCC SHRINKDB('MicrosoftIdentityIntegrationServer', NO TRUNCATE)
maybe preset size if you can predict
Log file should be 1/4 of total data file
Transaction Log is key to performance as well


Recovery Models
Full - LOG EVERYTHING
Simple - only as good as last full or diff
Bulk Logged hardly ever used with MIIS - only good for bulk logged operations

Red Gate SQL Backup to compress backup files (compression eats CPU time but far less Disk IO

When you clear run history it will host your transaction log. Do this during a maintenance window and
a. Truncate log
b. Change to SImple Recovery Mode
c. Clear run history in small chunks
d. Then go to Full Recovery mode and Perform a Full Backup

you can also add some indices to better performance

indexdefrag on DB every month, index rebuild quarterly till SQL 2005 EE. With SQL2005EE we get to rebuild index with DB online as well as rebuild index in TempDB
http://www.sqlbestpractices.com

Where's the WIFI?

So the sponsor of this conference is NETPRO. NETPRO as in Network Professionals. Their logo is KNOW YOUR NETWORK. Thus it is that I am at a loss to explain WHY?! WHY cant I get wifi in the conference rooms. NO ONE CAN! Turns out they've outsourced that to Cox. Nice.

Think Gloves

So the last two presentations I've seen have been technically brilliant. Things like Fuzzy Logic algorithms and using bit vectors and GUIDs to derive group memberships in AD, etc. Here's the thing, both solutions are geared to companies wanting to take shortcuts with their problems. I'm baffled why those companies don't spend more time crafting proper simple solutions to their problems as opposed to coming up with an overly complex solution.

The first was a means of provisioning access to non-AD based applications (applications that cant leverage AD for authentication) based on AD Group Memberships. The problem was notoriously difficult to solve and as it ended up, they crafted an XMA based on AD Groups using MVGuids and the bit vectors from AD which are used to denote group memberships. They then provisioned this in the applications in question. Brilliant right? Lots of code and very fancy terms. Then I thought, what was the real problem? Why manage access via AD Groups to an application that can't natively talk to AD? Well most companies don't have a website to allow them to request access to roles so the Tech Support teams manage Roles (see Groups) via Active Directory. The far simpler solution would have been to modify the applications in question to query AD directly rather than create new XMA's for every group in AD.

The fuzzy logic presentation was equally bright. There was lots of code and pattern matching with confidence scores, etc. All to allow non unique field to match up and create solid joins. There's obvious appeal to the idea, we could have used it for increasing the reliability of the name matched between HR and IDMGMT. But here's the kicker, why not create a GUID to match all the entries on both sides? Again, I think of the next version of something really complicated and then I stop and I think to myself...GLOVES.

The only positive outcome of all of this is that Apollo seems to have been very fortuitous in its software development leadership and wise in its choice of solutions to thorny problems. We remain light years ahead of the competition. We should seriously consider selling our solution or at least consulting with companies headquartered in sunny, tropical seaside locales.

Finally the meat

After being nauseated by yet another vaporware Microsoft presentation, BMC gave an excellent overview of how to use XMA's to do things like provision OpenLDAP, MySQL, and ORACLE.

Privileged access project? Done. Now the only challenges are really getting the access to do the actual provisioning and getting the list of servers to maintain. One small wrinkle, XMAs seem to be based on one off connections so we'd have to run quite a few XMAs to cover all of our Oracle databases. Still, good problems to have compared to I cant do it with MIIS.

Kim Cameron's Keynote

Seems like he is making a case for the OpenID movement. So, point one, we're becoming more and more interconnected. Businesses will do more and more business digitally, hence the need for agreed upon standards for communication, interconnectedness, etc. This will result in “de-perimeterization”, the dissolving of corporate boundaries, firewalls, routers, etc. He's advocating “legonic” (Lego like) technologies and businesses where they can easily be connected. Tomorrow's systems will be “agile and self organizing” and good at handling multiple sources of information with variable credibility. Cameron's point is that just like Remote API fell to the wayside in favor of loosely coupled Web Services and SOA, so to will identity management.

Enter in CardSpace and OpenID. This allows people to make “claims” about themselves or others and then to set up infrastructure to allow independent third party providers to validate the claim. This is EXACTLY the same point that SXIP's founder Dick Hardt made two years ago in his presentation to Open World (although his use of the Lessig style was far more compelling). Cameron also postulates, as a side note, that there will be increasing regulation around the concept of identity and anonymity on the Internet and the interconnected world.

So Microsoft's work is geared towards building out Cardspace to interact with things like ADFS, Security Token Services (STS), and Web Service Security protocols. Questionable? I wouldnt know, there was NO question and answer period provided for Mr. Cameron's keynote. Is he asking us to blindly validate his “claims”?

Microsoft to the...oops BSOD

Well well...no shock that the show came to a close when the Microsoft rep got up to speak. Showing off ILM2 (with some admittedly cool features) caused the Virtual Server (Microsoft's competition to VMWare's ESX server) to crash. Of course he was using Vista and getting messages ALL THE TIME.

The real kicker was knowing that all of this was
a. Not available until Mid 2008 (when did Microsoft ever miss a ship date)
b. Gonna cost us an arm and a leg given its coolness.

To boot, the guy doing the presentation was an unkempt retro hip version of Bill Gates with a bad accent. While I love Bill he's not selling me anything.

So far so good

Advanced MIIS training is good but its not the 504 course (Advanced MIIS). Its new technologies and it is COOL. Some items to get excited over?

XMA's: Imagine you want to connect to a MySQL database with MIIS. Normally I'd say you're SOL (look it up yourself). NOW, with Extensible MA's (XMA) you can write your own MA connection to it and then leverage it in MIIS. Web Services not a database? No problem. Centre Vu id's? Have an API we'll plug it in...

PowerShell: Almost had the big O on this one. Imagine a Cygwin/Ruby/Shell for Windows Servers....Now imagine scripted .NET where EVERYTHING is an object. Now imagine Kevin Feingold or Eric Treeman turned loose on this bad boy. IT IS SWEEEEET! We're going to retire the CMD files and Launcher.exe and go with the Power Shell scripts.

Advice: We're always really scared of putting bad data out there...Oxford's advice? NO BAD DATA in the MV. No garbage in....no garbage out. Lots of work on validation of data in and validation of Brad Turner's advice.

Gotta run...self service password reset is on....