CircaSpecting: People, Tech, and Business

Of Card Hustlers and Tech Sales

“Let me show you what our product can do with your data.”

Ugh.

Don’t get me wrong, I love data analysis as much as the next guy, and our product has great features for slicing/dicing/ computing/ visualizing data.  But, I cringe when a customer fixates on that sort of demonstration.  Because, as an industry we’ve been doing this computational and spatial and logical manipulation of numbers FOR SEVERAL DECADES.  Let’s just say, as an industry, that we’re 95% of the way “there” -- meaning we’ve achieved a 95% perfect calculator.  Most of what’s new now is nothing more than blinky lights, shiny things....a neat parlor trick.  

So what.

If you want to make a meaningful improvement in your organization’s effectiveness, efficiency, and creativity, you need to look for a 10X improvement in something...or a 100X improvement in something.  

0.05X is just noise.

Whatever is keeping your organization from being more effective, efficient, and creative....IT IS NOT A CALCULATOR PROBLEM.  The solution is not a better tool to allow a user logically, mathematically, or spatially to manipulate her data.

What should we be talking about?  

There is a growing body of recent organizational research demonstrating three of the biggest issues inhibiting knowledge workers:

1. They don't know what data are already available on the subject, or cannot find it, or cannot understand it
2. They don't know who has expertise in the subject, or who is already working on the same subject
3. They don’t know what’s been tried before, what was experienced, and what was learned

Notice a pattern?  Ignorance.  After 50 years of calculator investment -- LITERALLY TRILLIONS OF $ -- most employees are ignorant: they are unaware of the information assets, human assets, and corporate experience THAT ALREADY EXIST IN THE ORGANIZATION.  There’s a good argument being made now that the degree of ignorance on these dimensions is something like 99%, which means that there’s a 100X improvement opportunity here.  Sound high?  Consider this: knowledge workers and managers report spending more than 25% of their time -- every day -- searching for information they need.  Every day.  And, here’s the real killer: that’s just the time they spend looking for things they know to look for, things they believe there’s a chance they’ll find, etc.  There’s a radical step-function opportunity in efficiency, effectiveness, and creativity lurking in the knowledge-sharing problem.

So, please, please, please.....stop asking to see what a product can do with your data.  You’re falling for a card-hustler’s hypnotic tricks.  Keep your eye on the frontier of 100X improvement, something that can actually have a noticeable impact on your team’s results: 

1. Making sure employees can find and understand the data available across the organization
2. Making sure employees can find and engage expert coworkers across the organization 
3. Making sure employees can learn from the community’s experiments and experiences

 Once again: it's all about people.

Fusion-powered Idea Machine

Collaboration is not additive, it is multiplicative.  

This sounds sort of math-geeky, but it’s actually some pretty simple logic – and it provides a very nice scheme for thinking about how a group is more than the sum of the individual members' knowledge.  This is the magic of collaboration, when the fusion of our knowledge produces something more.  

Here's how it works.  Aside from his/her computational and creative skills, each person in the team brings two things to the party.

E is the explicit knowledge of a person, expressed in his/her artifacts.

T is the tacit knowledge of a person, expressed in his/her intuition.

To keep things relatively simple, let’s focus on a team of two people, a and b, each of whom carries both types of knowledge discussed above.  

a = (Ea + Ta)        b = (Eb + Tb)

When they work together on a problem, sharing both elements of knowledge with each other, some powerful idea cross-products emerge – activating Tacit knowledge in lots of places:

(Ea + Ta) x (Eb + Tb) = EaEb + EaTb + EbTa + TaTb

The first term EaEb is combining artifact-based knowledge. Left brain + left brain.

The middle terms EaTb and EbTa are the intersection of one person’s intuition with another person’s artifacts. Left brain + right brain.

The final term TaTb is the collision of multiple vectors of tacit/intuitive power — it’s the magic that happens when we are both at the whiteboard together.  Right brain + right brain.

One caveat: as discussed in the sidebar above, “activating the Tacit” requires a degree of interactivity, experimentation, adaptation, sampling, etc. that is technologically difficult to deliver in an appealing user experience.

Therefore…

Based upon the framework above, here’s a representative list of capabilities your collaboration platform must provide if it is to start the innovation reactor: 

Sharing (EaEb, the first term)

1. Upload, share, link, embed any artifact, data, widget or feed from any source
2. Multi-dimensional organizing  and associating with tags, keywords
3. User-defined relationships and links between artifacts
4. Integrated search engine for all formal and informal knowledge
5. Mesh trust model for fine-grain sharing

Extending (EaTb and EbTa, the middle terms)

1. Rich, evolutionary user profiles including endorsements and expertise
2. Emergent group/room/discussion formation and feedback solicitation
3. Conversations (email, IM, voice, vidcon) associated with artifacts & vice versa
4. Direct commenting, textual annotation, and margin drawing by colleagues
5. Interactive sampling, remixing, derivation from all shared artifact types
6. System recommendations based upon content, tags, and user behavior 

Creating (TaTb, the final term)

1. Private communication channels and pre-publication experimentation spaces
2. Ability for users to synthesize data from any source easily, impulsively
3. Simultaneously co-created digital whiteboards, spreadsheets, documents, charts, presentations, Gantts, etc.

What is Collaborative Intelligence?

I get asked often to define this term of Collaborative Intelligence that we started using a couple of years ago.  As I’ve said on my blog, I am a little wary of labels and categories.  So, I’ll simply convey what I see converging in the real world: Collaborative Intelligence is a set of technologies and techniques helping groups of people apply their collective intelligence toward making good decisions together.

It’s important to remember that every hour of every day at work, every employee is making decisions.  If you are a knowledge worker – actually, even if you hammer rivets or crush rock or solder wires for a paycheck – then the decisions you make are (hopefully) dependent upon something like the following Decision Stack:

• Data:  the set of available facts
• Model: a theory or algorithm about how things work
• Aspiration: preferences, desires, or goals and their rationale
• Compromise: tradeoffs and diplomacy among competing interests

Every little decision we make each day involves this stack. Sometimes certain pieces of the stack are flawed; sometimes they are subconscious; but, all decisions involve these things.  And, every day, all day, we are doing this at work.  “Collaborative Intelligence” is just a short-hand label for what we see when we are working together to make better-informed decisions.  So, it’s about how groups of people (2-n) explore data, models, aspirations, and compromises to help make things better.

 Collaborative Intelligence is not just some new features added to Business Intelligence.  Business Intelligence (BI) is a field of technology that has made great contributions over the last 30 years in the preparation, storage, retrieval, and distribution of facts.  To a meaningful but lesser extent, it has also made contributions in the area of developing  explanatory and predictive models.

Unfortunately, mainstream BI has gotten stuck in a corner in the above diagram, relying on technologies and techniques that are fundamentally centrally-driven, static, data-focused, and impersonal.  This has opened a great door for more grass-roots, participative, people-centric, adaptive, edgy, conversational, and diplomatic tools and techniques, like what we see evolving in the Enterprise 2.0 space.  The grass-roots conversations and activity streams that are the stock-in-trade of Enterprise 2.0 platforms are full of information-scent, Pirolli’s concept of little hints about hidden-but-interesting things.  So, when using those platforms one feels naturally drawn into conversations and topics and investigations, within which one can share additional facts, models, aspirations, and compromises. 

So far, the Enterprise 2.0 tools have not addressed the important BI territory of Data and Models; but, it is only a matter of time until they realize that collaboration without Data and Models is just idle chatter.  It is up to the BI community to realize at the same time, however, that People are essential to the process of applying data to make things better.  I have no doubt we will see the tools and techniques of BI expand along the lines indicated by the arrows in the diagram above, just as the tools and techniques of Enterprise 2.0 expand in the opposite direction.  This convergence will be wonderful for users.

This is what we are working on at Lyzasoft: helping people find the information and experts they need to make better decisions.

D-day Plan for Enterprise 2.0

If you want to invade, you need an plan.  And, you have to take the beach.  I believe we are on the cusp of Enterprise 2.0 adoption, and I believe the analytical community will be the beachhead where collaborative platforms begin to get traction within the enterprise.  Why?  Because the activities of Enterprise Web2/Social are compatible with the activities of Enterprise Analytics, and because their tools and technologies are complementary.  This post outlines a framework for how to blend BI and Web2 to create such a beachhead.  I want to open up discussion on how the technical components of each domain fit together to create a platform for Collaborative Enterprise Intelligence.

The diagram below has three sections.  First, the center stack shows the stages of information processing in COMMON TO BOTH the Web2/Social domain and the BI/Analytics domain.  The dotted lines show the recursive circulation that is the information life cycle.  Second, the left edge shows three concerns central to BI/Analytics.  Third, the right edge shows three concerns central to Web2/Social.

Let's define the terms used in the central stack, starting at the bottom -- where the process starts:

• Synthesis:
  Finding, collection, and integrating information of interest from a diversity of sources
• Analysis:
  Deconstructing and experimenting to compare/contrast, diagnose root causes, and forecast trends
• Publication:
  Packaging and distributing information capital for broader/easier use
• Adaptation:
  Refining published information capital with new/additional/different context information
• Leverage:
  Systematically applying the community's best information assets in all contexts where relevant

Whether you're coming at the question from the BI side of the house or from the Web2 side of the house, I'll bet you generally see the framework of your world in that stack.  You may be using different words, but you see probably see familiar substance.

The similarities are exciting, because they mean each side has a potential ally to achieve its objectives within the enterprise.  For the BI folks, this alliance could crack the code on how to expand the penetration of BI use and involvement beyond a small cadre of power users.  For the Web2 folks, this alliance could get your foot in the door to show enterprises how social/collaborative tech can add value to existing processes and technical investments.

But, before we jump in, there are a few more pieces to be reconciled.  The outer edges of the first diagram show some central concerns each ally brings to the partnership negotiation.  The BI camp will insist that the blended platform support Traceability, Manageability, and Security.  The Web2 camp will insist that the blended platform support  Trust, Collaboration, and Personalization.

• Traceability:
  See where the information capital (data and logic) comes from and where it flows
• Manageability:
  Avoid overwhelming the IT staff with infrastructure management, policy enforcement, and user support
• Security:
  Manage access to and participation rights within the community
• Trust:
  Fine-grained, independent sharing decisions made by each member for his/her work product
• Collaboration:
  Serial, parallel, simultaneous, and composite means for multiple members to contribute to a solution or conversation
• Personalization:
  Experience (UI, data, channels, etc) that is tailored by the member, and a zero-anonymity environment

Here, around the edges, I think some challenges remain.  They are not insurmountable, and they are not isolated to one partner or the other.  Again, the good news is that we are in this together.  First, there are things that each side must do to get its own house in order. 

For the BI folks, we need to accept that despite our Traceability speeches, we still have relatively limited lineage tracing in our analytical platforms; users pulling charts and reports cannot see where those things came from or how they were produced or by whom.  And, no, it's not sufficient to say that the DBA can pull up his/her SQL scripts if we're interested.  Members of the intellgience community need the ability to answer FOR THEMSELVES the age old question: where did that number come from?  Today, they can't, and that's a problem.  We as a BI profession also have not cracked the code on Manageability.  If you doubt this assertion, please consider that percentage of the numbers used by your analytical team (all of the team, not just your BI power users) are inside the BI tool vs. what percentage are inside spreadsheets and desktop databases.  When something is out of sight, it is clearly not manageable.  BI need solutions that bring more of the analytical work of the community out of the gray market of spreadmarts and into view.  Until we offer compelling solutions that bring most of the activity into the fold, we cannot complain that Web2 is out of control. We can solve these problems, and in fact there are tools on the market today that can help you get there.

For the Web2 folks, we need a workable Trust model for the enterprise.  Our coarse-grain trust models simply do not fit the complex relationships in the enterprise world, where the dynamism of special projects, insider information cycles, compensation programs, stealth go-to-market plans, and the like mean that Bob needs to articulate a very precise specification of his trust of Sally.  Again, this problem is not insurmountable; models and products exist for solving this problem.  We just have to make it a priority.

BI and Web2 are made for each other.  Web2 can grow BI's reach significantly and quickly.  BI offers Web2 a ready subculture of users who already practice and value collaboration (albeit in the gray market, outside formal platforms).  Experts from both sides need to work together to blend the best of both practices into a seamless experience that can showcase the power of Enterprise 2 for all types of organizations.

Please feel free to use, modify, extend, rant, rave, suggest changes to, link to, talk about, or forward these ideas on the necessary capabilities and how they can be organized to deliver Collaborative Enterprise Intelligence.

Paths to Pervasive

I just returned from a Business Intelligence (BI) industry conference where everyone was talking about how to make BI pervasive.  That is, "how do we get every employee to be a user of reports, dashboards, and analytics?"

You don't.  For 2 reasons.

There are actually very few product categories in the world that can expect to be needed by everyone.  There are people who do not need a car, a mortgage, a gun, a college education, a smart phone, a telescope, etc.  Similarly, there are people (many of them) who do not actually need BI stuff.  Sure, BI will grow from its current low penetration -- but, let's be realistic and set the bar lower than "everyone's a BI user."

Also, you've got the wrong pronoun.  We the technology/solutions/services providers will never have the time, influence, credibility, or relevant experience to connect with all the potential users of BI reporting and analytics.  The direct strategy is a dead-end.  We have to take the indirect path: equip and recruit influential BI users to build their own user bases.   In other words, we have to use a channel strategy within the organization, just as we often use a channel strategy externally to reach hard-to-reach market segments.  To do this, we have to enable interested BI users to create a personal brand, to promote their info-wares to downstream subscribers.  This requires that we reconsider the boundaries of BI: it is not just ETL, warehousing, query, and formatting.  We must make BI integral with everything required to facilitate users creating their own info-brands, which certainly includes (but is not limited to) social media, widgets, search, collaboration, etc.  Then they will promote the consumption of BI info, albeit not necessarily in a form that we would recognize as such today.

The useful question is not "how do we dumb down BI tools and sell them to everyone?" but "how do we turn core users into channels to reach peripheral prospective users?"

It's not about the tech.  It's about people and relationships.  It always has been.

Lyza: Spreadmart-free Data Analysis

Last week, I met with a couple dozen industry experts and journalists to do show-n-tells on Lyza, in advance of our official launch on September 22.  Lyza, in case you missed my earlier blog hints, is our new desktop software for data integration and analysis. 

Anyway, after several years of research, design, and engineering work, you'll understand why I breathed a sigh of relief when....they liked it. 

Here is a sampling of what they've been writing about Lyza:

all-in-one data analysis tool for the desktop ...well thought out ...highly graphical ... very wysiwyg

an easier, more intuitive way of pulling data together from diverse sources, combining and manipulating it and creating results and reports for distribution

well suited to analytic needs...a graphical environment allowing analysts to build up analysis iteratively...access to most data sources...auditability

Read some of the articles via these links..

Barry Devlin, BI/DW author and former IBM Fellow
http://www.b-eye-network.co.uk/blogs/devlin/

Andy Hayler, CEO of The Information Difference
http://www.it-director.com/technology/data_mgmt/content.php?cid=10744

Shawn Rogers, Editor at The Business Intelligence Network
http://boulderbibraintrust.org/brain_trust_blog/2008/09/lyza-software-launches-at-the.php

Now, try it yourself, with a free trial download from www.lyzasoft.com.  And, spread the word!

Imagination is Elastic

The bigger the space occupied by your product -- the more specific and detailed and tangible and exhaustive (exhausting?!) your description of what it will do for your customers -- the less room is left for ... the customer.  For his or her imagination, for the unique mark, for the creative -- for identity.  Imagination is very elastic; it will contract to nothing if you leave no room for it.  (Kathy Sierra wrote a great post on this topic, here.)  No matter how wonderful our product’s features, we have to remember: it’s the imagination of the user that binds the features, holds the whole together, and makes the product come alive. 

Recently we've seen this demonstrated powerfully in our user labs, when people use features in ways we don't expect -- extending the definition of the product itself.  If we dictated to the lab participants exactly how to use something, we'd probably get a pretty boring regurgitation of our own views. 

So, our central challenge is to recruit the user’s imagination to the task of creating our product from the features we’ve engineered.

When products falter, the missing ingredient usually isn’t a feature; it’s IMAGINATION ... as in, the users'.  Make room for it.

Then with a pleasure which is both sensual and intellectual we shall watch the artist build his castle of cards and watch the castle of cards become a castle of beautiful steel and glass.  Nabokov

Usability, Early Adopters, and Crossing the Chasm

We've spent a great deal of time and money this year running observation labs to evaluate user cognitive processes, toward the goal of understanding how to make Lyza (www.lyzasoft.com) software that is both powerful and easy to use.  While this research has been invaluable, it also contained a trap we did not foresee.

Initially, we thought that Early Majority users would express higher expectations for usability than did Early Adopters.  But, they did not.  Their expectations for good usability were almost identical.

At first, this puzzled us and made us wonder if we were perhaps ready for the hockey stick of the mass market.  (Given the optimistic psyche of entrepreneurs, this trap is particularly dangerous.)  Then, we noticed something else -- Early Adopters showed significantly higher tolerance for a gap between their expectations and the product's actual usability.  EA's were just better able to get past, work through, or work around perceived deficiencies than were Early Majority users.  Fortunately, the labs allowed us to go beyond simply asking about expectations and allowed us to observe the differences in behavior and tolerance. 

Many new tech projects see a stabilization of expressed expectations at some point, encouraging them to scale-up marketing of their products in hopes of tapping the exponential growth curve in the first graphic.  But, they jump in without the significant additional investments of money and staff required (see second graphic) to improve the product faster than expectation-performance gap tolerance is declining in their target market (see "The Cliff" in the first graphic).  I think these conclusions are simply another perspective on the now-classic Crossing The Chasm argument: don't count your money based upon Early Adopter response, because there is a lot of work left and a very tight window within which to do it.  As a professor of mine once said ...

Growth consumes large amounts of money and staff.  Have stockpiles of both before you start.

The consequence: we need to hustle to grow our Lyza engineering staff ASAP.  So, if you're an expert Swing developer, send me your resume.

Software and Dancing?

Sorry I've been so quiet the past few months.  We're launching a new software venture (www.lyzasoft.com, shhhh -- we're still in beta), so I've been balancing two jobs. 

Over the last couple of months at Lyza, we've been busy with some intensive research programs focused on software-user cognition, which I'll talk about in some posts next week.  Though we've been driven heavily by secondary research in this field for the last couple of years, our own recent primary research has allowed us to watch and get-inside-the-heads-of dozens of analysts working through their typical challenges of gathering data, analyzing it, reporting on it, etc. 

And, I can say with some certainty that William Butler Yeats had it right:

How can you tell the dancer
from the dance?

Analysts' work defies tidy boundaries like Business Intelligence, Data Integration, Desktop Productivity,   Data Warehousing, etc.  Technologists and strategists spend a huge amount of time and energy debating ideal dance moves ...

How will Search work within Business Intelligence? 
Should Data Integration products enable SOA? 
How can you deliver reporting without a Data Warehouse?
What's wrong with spreadsheets?

...without actually slowing down, watching, and absorbing the real people performing the dances.  When you see the dancer and the dance as inseparable, those arbitrary technical categories blur -- and you realize that the REAL cast of people involved in data integration, analysis, reporting is FAR, FAR larger and more diverse and more dynamic than you imagined. 

A Simpler BI Scorecard

Ben Schneiderman does an excellent job of classifying the tasks in the creative, analytical, innovative (aka “thinking”) process in his book Leonardo’s Laptop. 

Since, theoretically, the purpose of Business Intelligence is to feed people’s thinking, Ben’s framework seems like a perfectly simple scorecard for evaluating BI tools.

What follows are my assessments of the standard BI tools along Ben's criteria:

F    Searching for data
F    Visualizing data and process, to discover relationships
F    Consulting with peers
F    Thinking by free association to create new combinations of ideas
F    Exploring via what-ifs and simulations
F    Composing artifacts and performances step by step
F    Reviewing and replaying to support reflection
B    Disseminating results to gain recognition and to support the community
F    Allowing for all of the above steps to happen recursively and iteratively

It’s easy to see why standard BI tools are not the solution for the Thinking Caste.