Saw an interesting comment in Southwest’s SPIRIT magazine (April 2011): “Creation is pleasure and torture at the same time. It’s trying to make something out of nothing. It’s a birth of some sort. Sometimes it’s like pulling a piano out of a swamp. Sometimes it’s like walking on air. The torture of that nothing-space in front of your, and the pure elation of filling that space with something good — it’s one of life’s great juxtapositions. I am grateful for that — the torture and the pleasure.” Now Zooey Deschanel was talking about acting — but it struck me that it also applied to one of the things we focus on at RCS — the creation of a new decision support idea to fill a need. And that the feeling is very similar, after all our CSE analysis, you sit facing a blank screen, or a blank whiteboard, or a blank sheet of paper, and have to initiate the design phase…and create a new idea. At RCS we take that step seriously, and use a variety of Innovation Practices as part of that design process. By adding innovation to CSE, we are able to fill that ‘empty spot’ with creative ideas inspired by the domain analysis from ACSE(A), by the lessons learned from other domains with similar Joint Cognitive Systems characteristics, by design patterns that have proven appropriate to decision similar situations.
I commonly get asked by the new Cognitive Systems Engineers why they are struggling and frustrated to ‘figure it out’, but ‘just cant quite seem to make the breakthrough’. When I give my enigmatic answer: “that frustration you’re feeling is a good indicator…its the feeling that always happens when you’ve done the analysis, the KE, the background research…and that good idea is churning around in your subconscious…its a sign that good idea is just about to occur”. They rarely accept it, nor understand it…until it happens to them a few times…then they learn that ‘agony’ is foreshadowing the ‘ecstacy’ of designing a revolutionary, breakthrough decision support system.
We recently gave the final presentation to our customer(s) on our system Usability evaluation; customer(s) is notionally plural since we presented to all the different perspectives - users, managers, financial managers, developers, contractors, etc. This combination makes tailoring the presentation difficult, so we just stayed with our ‘center’: insights about the current system inspired by Cognitive Systems Engineering.
By this point, the customers were becoming familiar with our unusual point of view, and the resulting unusual insights. They had also raised their expectations of what would come of all this. Initially they expected ‘look and feel’ improvements…now they were demanding a truly more effective system, one that would dramatically impact the mission user’s success.
The report itself was quite extensive, over 140 pages, containing everything from improvements to menus to complete re-working of the display navigation scheme. Some of the highlights:
1. By functionally re-designing the workspace (the screen layout of the windows), a work-centered functional process model was imposed on the system. Users naturally saw an ordering (and more importantly the functional coupling) of the mission work. Indirectly, it also ‘pulled together’ features that had been sprinkled throughout the existing environment.
2. Individual windows were redesigned to ‘encapsulate’ the support for work decisions, resulting in a more ‘compact’ overall system. The number of independent windows was reduced approximately 8-fold.
3. The improvements in 1 and 2 together resulted in a system that was much easier to operate. By redesigning the tool to have much less ’system effort’ left more user energy for the ‘mission effort’. The customer(s) saw that immediately.
4. Additionally, the improvements in 1 and 2 together implicitly identified some critical ‘work gaps’ that weren’t supported by the existing system. We provided some interim, initial concepts to fill those voids.
By the end of the evaluation, we had actually identified the significant functional capabilities that had been lurking under the hood of the existing design. We actually became enthusiastic about their potential for mission support. Our recommendations represent some pretty significant rework of the interface software and some of the underlying processing and persistent storage of the systems. But the developers (and even the bill payers) were enthusiastic to produce the next version of the system with those enhancements.
From the initial request to “make it look better so users will accept it” grew an insightful, deep analysis of this system artifact, supporting real mission work, for real users - the essence of Woods’ Cognitive Systems Engineering Triad.
After the recent DC Metrorail crash, which tragically took 9 lives, the standard flurry of reporting activity followed. Any time an accident like this happens (see the Air France crash before it), all the reporters and amateur investigators try to pinpoint blame – and the easy answer is almost always to identify the culprit as human error at a single point of failure. However, one recent article looked a little further and began examining the problem from a more powerful point of view – taking the same human-centric approach to systems that we do at RCS.
Mr. Vedantam of the Washington Post explored the problems that come with introducing automation into systems that humans have traditionally controlled. He correctly states that the problem often comes from a poorly designed relationship between the human and the automated system. In fact, the successful operation of ANY system depends on the relationship between the human, the artifacts the human uses, and any software agents (aka automation) that play a role in operation (known as a Joint Cognitive System). This relationship must be properly supported in all systems, not just when interacting with automated systems. Designing from this perspective, which is what the ACSE approach is about, drastically changes how system designers approach designing systems to support users. This approach forces designers to consider the implications of their design decision on the whole system, rather than on the construction of the individual components.
When adding automation, the major implication of this approach is that the human user is not replaced from the system, as some designers want to believe, but instead switches from operator to supervisor. In fact, in some ways the demands on the human *increase* as they now have to know their original role, but also now become expert in how to validate the automation’s decisions and how to interact with it. Automated agents are essentially another form of operator, and thus have to be team players. Users should never be told to not interfere while being expected to be the last line of defense for proper operation. That means users need to have the same information that the automation has to make decisions (and presented in a manner that the user can quickly understand and make decisions from), just in case the unexpected happens and the automation isn’t able to perform. This is essential especially in situations where the unexpected happens, since it is the human ability to think and react in unexpected situations where human judgment and action is unmatched – something no level of automation can ever replace.
During these times, when the user only has seconds (or less) to react, the overall system design must come through. Good informative feedback is important, as is a clear representation of what the automated system is doing, so the user can quickly step in and take the right actions. For example, the user must know when the automation is nearing its operating limits, or when the automation has failed (as it did in the DC Metro crash), so that they can switch back from manager to operator without the overall JCS skipping a beat. This is a lesson spoken frequently throughout the history of Cognitive Systems Engineering (see Lützhöft and Dekker, 2002, for an in depth evaluation of the problem found during the Royal Majesty incident mentioned in the article, where automation failed and the user was unaware), and it appears that others are catching on as the addition of automation has continually failed to lead to an accident-free world.
If no one takes responsibility for designing the JCS as a whole, it is always the human user who has to deal with the consequences. And sadly, they are almost always burdened with the blame. Taking into account the CSE perspective, as Mr. Vedantam has, hopefully that onus will be shared with system designers to build a better JCS.
Lützhöft, M. H. and Dekker, S. W. A. (2002). On Your Watch: Automation on the Bridge. Journal of Navigation, 55(1), 83-96.
It’s interesting trying to introduce Cognitive Systems Engineering (CSE) to a new customer meeting (a mix of users, managers, and system developers). In some ways it’s a bit like speaking a foreign language. Terms, concepts, patterns, etc., that are fundamental to us are like puns…they are almost impossible to translate. (I once saw a James Bond film with German subtitles…the translation was accurate…but the puns were gone…the audience thought I was nuts for laughing.) We always get a bit of that when we talk CSE to a new crowd, particularly for something like an evaluation…where they are expecting things like “change font to Arial 12″ or “change this green to brown” and get things like “the workspace doesn’t adequately reflect the underlying domain workflow”.
System developers on many of our other projects are flocking to Agile Development as a way to ‘ensure user acceptance’. The premise is ‘build something’ (almost any design will do) and the users will then try to use it, gripe about it, and iteratively guide the agile team toward converging on a ‘user accepted system’ (assuming that to be synonymous with good Decision Support…but that’s a blog for another day). In this evaluation project, our design methodology is essentially turned inside out, into an evaluation methodology. The initial insights presented to the customers went remarkably well, largely because while we used Cognitive Systems Engineering as the basis for our approach, the resulting insights naturally lent themselves to being expressed from a user’s perspective. With the insights being spread across Usability (the standard ‘knobology’ kinds of comments), Usefulness (efficiencies of just getting common work processes done), and Understanding (related to truly effective decision support for fundamental problem solving…critical during novel, unanticipated situations), we were able to ‘ease’ them into the evaluation findings to date. We opened with the Usability ones, and the discussion naturally broadened to the larger impacts on the user when these flaws confused them. We then transitioned to the more abstract Usefulness and Understanding findings. These more holistic, abstract findings were tough to discuss in depth…but the magic happened when we started showing design recommendations. We call it the “of course test”…once customers see it, they blurt out “of course it should work like that!”
The first review meeting, surprisingly with findings at all three U Levels (Usability, Usefulness, Understanding), was a little slow in starting, but really gained an acceptance for Cognitive Systems Engineering, not so much for its underlying science, but more for its ability to detect issues deep within the conceptual design of the system and offer direct resolution to them. The redesign recommendations are what made Cognitive Systems Engineering valuable to the users, managers, and developers of the system.
Next up: Part 4: Second round of Insights-A Case Study in Cognitive Systems Engineering Usability Evaluation
We’ve learned (the hard way) that many of the systems we’re asked to redesign, evaluate, or replace have (in our atypical Cognitive Systems Engineering point of view) such significant, fundamental issues that we now start with what we call ‘Analysis by Inspection.’ Essentially, there is no reason to conduct expensive and intrusive observation opportunities (such as simulator tests, etc.) until the larger ‘rough edges’ can be knocked off. This is a bit unorthodox for many customers who expect a hoard of clip board carrying evaluators instantly descending on their users. By ‘Analysis by Inspection’ we mean taking a step back from the system and ‘looking at it’ from a Cognitive Systems Engineering perspective. It’s an extremely productive way to discover the ‘low hanging fruit,’ but it also allows an initial assessment of whether the system is ready for more in-depth, but much more expensive. evaluation approaches.
For this particular evaluation our initial efforts are going well. The startup activities included (for example) a full day train up session (described in Part 1) and reading the user manual. Our Analysis by Inspection compiled some of the typical information:
- This system is composed of over 100 individual windows, popups, dialog boxes, etc. (The developers were surprised by this number as their initial estimates were much smaller. The system uses a mix of buttons, tabs, menu bars, pull down menus, etc. (not unique or surprising), but we immediately noticed some ‘clustering’ of the 100 displays and the usage of various controls: some favored tabs, some favored buttons, others menus.
- Between 100 and 200 controls, a combination of buttons, icons, mouse actions, moused menus, menu bars.
- The ‘older’ parts of the system showed evidence of incremental, agile development, sometimes having four different ways to navigate to the same state (a classic sign of a ‘literal’ style of User Centered Design, meaning at some point in time, a user said ‘I want to be able to get from here to here’ and the developer did exactly that, rather than consider a larger context of the design of the overall system.)
Much of the work of this ‘Inspection’ was captured as sketches and notes on the Project Wall for this effort. The system was ‘disassembled’ (picture a computer system version of TLC’s Overhaulin‘) where much of the statistics above began to jump out, and some interesting evaluation insights occurred:
- The ‘disassembled’ system components were linked back together into sort of a navigational State Transition Diagram. User actions on controls changed the ‘display state’ of the system. As soon as you step back from that diagram, complex clusters of paths become apparent (like a complex part of a city street map). Even more interesting are where the developers responded to user demands by installing menus of top level ’shortcuts’ (our explanation, yet to be confirmed). These shortcuts jump deep into the diagram, at the cost of damaging any Mental Model of the System being formed by the user. From a Cognitive Systems Engineering standpoint, we normally try to design the system to overlay on the Mental Model of the work domain itself, so the user doesn’t have to understand the work and extend that mental model with a second, unique one of the system (not even factoring in the complexity of the interrelationships needed to use the system to accomplish work in the world.)
- The system formed itself into clusters from several dimensions: a) density of ‘action paths’ linking the displays within the cluster (and a corresponding paucity of links between these clusters). Our unconfirmed assessment of this indicates separate development ’scrums’ over the life of the program–we’ll be confirming this with the development team. It would be interesting if we were able to be ’system archaeologists’ and rediscover its genesis by the fossilized evidence within the system. b) A difference of the interaction choices across these clusters (preference for tabs vs. menus vs. buttons.
- Several areas where ’system objects’ paralleled ‘world objects,’ meaning sometimes the user interacted with the basic object itself, sometimes with a system generated surrogate (typically linked by a naming text string). This required the user to remember which type of actions occurred on which side of this ‘divide,’ along with the differing command styles to operate on these two different classes of object.
The productivity of this quick “Analysis by Inspection” is evident on the project wall: the team used bright orange post-it notes to flag ‘hot spots’ in the system for further inspection. In a period of two days, over 50 of these orange notes and the associated ’so what’ descriptions of each were generated. We’ll be summarizing this initial assessment with the customer soon, and together making some decisions about whether some development should be done ‘to knock some corners off’ before we get into the more elaborate, expensive types of observations/evaluations.
Coming soon: Part 3: Delivering Initial Findings
Recently we were asked by one of our partner companies to provide a Usability Evaluation of one of their new systems. This is Part 1 of a series, to be expanded as the project unfolds. Hopefully it will capture a sense of how ‘different’ a deeply grounded CSE Evaluation of a system really is.
We started off by explaining (during the proposal process) that our Cognitive Systems Engineering background and our experience with designing revolutionary Decision Support Systems really ‘warps’ our assessments of systems. We see things far beyond ‘knobology,’ ‘look and feel,’ and ’style guide’ and see deeper underlying issues that fundamentally impact the success of the system, but are often much harder to fix, (similar to the idea that ‘Remodeling is more difficult than new construction’). Equally, we don’t believe that a Style Guide will cure all ills, nor do we believe one will prevent all future ills as the system continues to grow. After a big dose of “Be careful what you wish for,” we’re on our way to evaluating a system.
The system itself is a pretty cool GIS referenced, dynamic data repository, intended to support a team of users trying to analyze the data with a powerful set of geo-centric, direct manipulation tools. As a trivial example, ‘find all the data inside this area’ done by defining a box on a map, not by typing an SQL query. The whole project is further challenged by what Dr. David Woods calls “The Envisioned World Problem,” (i.e. the system they envision would radically change the operational practices once fielded, hence there is no body of expertise in working this way). That lack of expertise makes many system development processes impractical. In addition, the program has all the typical pressures to ‘build in an agile way’ (to give management lots of near instant gratification) and ‘build these features next’ (to give marketing new ammunition as they look for customers). Our planned approach is to rapidly construct a crude CSE Analysis of the cognitive work to be performed to ground our thinking. System design features, information presented, representational techniques, etc. will be evaluated against this rough ‘basis for support’.
To manage these support needs and findings, we use Woods’ categorization of Usability, Usefulness, and Understanding. Roughly stated:
- Usability: the actual operation of the machinery of the system (i.e. classic knobology, style, etc.) - successful system design on this level allows individual operations to occur
- Usefulness: the ability to ‘execute the workflow’, the everyday processing of information through the system–successful system design on this level allows routine work to be performed
- Understanding: the true, deep insight into the first principles of the work domain (and also the system itself)–successful system design on th is level allows the user to adapt to unexpected situations, to truly understand the world and the system
The parallelism to Rasmussen’s Skill-Rule-Knowledge levels seems readily apparent.
The project started with an all day training session (that we saw as an all day Knowledge Elicitation opportunity.) As the trainer described individual features, we watched for indicators. Since the training tended to be encapsulated feature-by-feature presentations, we expected to detect indicators related to Usability (since they intentionally were out of context of actual use). We observed places where the instructor operated a control, and didn’t get the expected result (literally “oops” or “why did it do that”). There were many instances of this class of observation.
Being the overachievers that we are, we began asking for more ‘Use Case’ examples, in the hopes of also getting examples of Usefulness cases. The indicators for these are different, this time focusing on ‘work processes’ and any interrupts, or ‘do-overs’ along the way. We gathered several very different indicators that triggered lots of questions and note taking, from “wait, let’s try that again” from the instructors to “now, XYZ should be in place here and we…wait, it’s not here.” While the language was feature/case specific, the ‘pause/regroup/retry’ behavior became easy to detect.
While just getting going with the entire process, one very interesting meta-observation came out of ‘Day One:’ by late in the day the instructors and developers were *self* detecting these indicators immediately after they occurred. They were correctly identifying the indicators, but also the class of the finding and also some powerful insights about the underlying system design causes. This System Remodeling Project looks like it will be interesting.
Standby for Part 2: Analysis by Inspection.
We had an interesting session with a customer the other day. The discussion centered on how Cognitive Systems Engineering ‘connects’ with the DoD Architecture Framework (DoDAF). After some interesting moments of talking past each other, the discussion settled on two main points:
- In order to be relevant in major DoD systems programs, Cognitive Systems Engineering will have to find a way to integrate itself into the DoDAF mindset,
- The Applied Cognitive Systems Engineering -ACSE(tm)- methodology (as an instantiation of Cognitive Systems Engineering) delivers revolutionary, high quality designs without the heavy overhead burden of the DoDAF processes.
ACSE was expressly developed to ‘fill in the gaps’ in IEEE Systems/SW Engineering processes. In essence, it was developed as a COMPLEMENT to such processes, not as a replacement. Internally, for relatively small scale systems, we find it produces powerful, revolutionary Decision Support Systems often ready for SW implementation. For some systems, ACSE gets augmented with other IEEE processes and artifacts (i.e. Interface Specifications (for data exchange APIs) and Software Requirement Specifications (SRSs) ). ACSE has also been worked alongside Rational’s Unified Process (RUP) with similar successful results. In short, ACSE delivers the powerful, revolutionary advances in Decision Support System capabilities that DoDAF, RUP, Agile, Extreme, etc. all lack, and depends on a quality development process to ‘deliver the goods’ in an actual working software system.
Many organizations adopt a process like DODAF and quickly lose sight of the objective. In such ‘artifact heavy’ frameworks, the artifacts themselves can become the focus, rather than the quality of the support being provided for the user/for mission success. The several times we’ve been part of a DODAF based program, the form and compliance of the artifacts overwhelmed any attempt to ensure adequate fundamental decision support was built into the system, each time resulting in a brittle Joint Cognitive System. In essence, these were cases of “building the wrong things well.”
It takes a pretty agile mind to make this all work out. Superficial understanding of DODAF fails to understand the inherit brittleness of its underlying Waterfall Methodology roots. Blind execution of RUP similarly loses sight of its spiral/object oriented biases. Taking a step back from these (and others) the broader context comes into view: these are essentially ‘construction’ methodologies that profit greatly when complemented by a generative process for discovering the key functional capabilities that are essential for the user to succeed with their work in the real world. Without such a complementary, user-centered process, ‘weak methods’ fill the void: RAD/JAD, XP, Requirements Decomposition, etc. But in order to integrate Cognitive Systems Engineering some ‘bending’ of the artifacts and task sequencing is required. As a simple example, the unique capabilities of proper support for Directed Attention has never appeared in any of the OV-x documents of a DODAF program. (Here, I hope we’re wrong and someone will offer one up.)
In summary, there is really no argument at all, it’s a case of ‘violent agreement.’ Cognitive Systems Engineering (in our case instantiated by ACSE) provides the breakthroughs in the necessary Decision Support to improve the robustness and mission effectiveness of the Joint Cognitive System. Systems Engineering processes are essential to constructing the computer (HW and SW components) of that JCS, and become increasingly critical as the scale and scope of those components becomes large. All it takes is to ‘allow’ the CSE insights to ‘enhance’ the various artifacts with good Decision Support System capabilities. Ultimately it’s about “Building the right things, well”.
In the March edition of Wired Magazine, two authors touched on Cognitive Systems Engineering in the course of their discussions of the current financial crisis. Daniel Roth wrote an article entitled “The Road Map for Recovery” about what it would take, in the way of strategic changes to rein in the financial markets and prevent the causes of the current market recession. The article focused on what it would take to inform investors adequately of the fundamental makeup (and risks) associated with the “products” they were investing in. Felix Simon wrote an article: “Formula for Disaster” discussing the causes of the collapse. The basic premise of the two articles was that if the true underlying data were available, and visual analytic tools were available, the wisdom of the masses would collectively be a more powerful force in detecting and revealing the underlying truths of the offering, helping would be investing make better decisions (and therefore not buy “CBO Squared Products” and Ponzi schemes).
In that article, the authors unknowingly rediscovered two fundamental elements of Cognitive Systems Engineering: Observability, the principle of exposing the underlying processing and transformations in a way that allows understanding and control; and Emergent Pattern-based Displays, the technique of design where the holistic meaning of the integrated data set reveals itself as patterns in the data itself, employing the perceptual system to detect this holistic meaning from the data presented in an intuitive frame.
In the articles, Roth wrote: “…the era of <data access> has to give way to the era of pixelization; only when we give everyone the tools to see each point of data will the picture become clear…“. This is*exactly* what has been discovered by the Cognitive Systems Engineering community. Filters, Goodness Scoring algorithms, relevance ranks, etc. are knowledge-sparse when compared to a properly framed Emergent Pattern-based Display coupled with the elegance of human perception and pre-cognitive processing.
Salmon also wrote: “…one reason was that the outputs came from ‘black box’ computer models and were hard to subject to a commonsense smell test….” Salmon was describing how systems without Observability are essentially impossible to fathom. The Cognitive Systems Engineering community goes further: users will invent ‘urban legends’ about how such systems work, how they respond to inputs, and how they ‘think’. These ‘urban legends’ tend to be very anthropomorphized and very rarely actually correlate to how the algorithms actually work, particularly for any complex, or non-linear mathematical formulation. These urban legends routinely collapse under atypical situations (since they were developed during ‘normal operations’) resulting in additional user confusion at the worst possible time.
Later in the same article, when talking about attempts to reduce the true underlying complexities to a single, ’simple’ metric (in this case loan correlation), the author states that “…co-association between securities is not measurable using correlation because past history can never prepare you for that one day when everything goes south. Anything that relies on correlation is charlatanism….” Yet traders built massive financial portfolios on exactly this correlation metric, even though it was a flawed simplification. They liked it because of its simplicity, but failed to have any Observability into its underlying brittleness.
In short, you find Cognitive Systems Engineering ‘case studies’ in a variety of places. In this case, studies in Observability and Emergent Pattern Based Displays are evident in the Wired Magazine edition about the current financial crisis.
Sitting on my desk is a cool little device that started this discussion the other day. It’s essentially a small, compiled neural net, wrapped in a cool plastic ‘eggshell’ that plays 20 Questions so well that users are ‘freaked out’ like it’s possessed. The discussion immediately turned to: “if it can do *that*, we can use it to solve <real world user need 1> and <need 2> etc.!” This is exactly the kind of thinking you’d like a tech savvy developer to experiment with; it’s also good when Cognitive Systems Engineers do it. The discussion came full circle back to “How do you know what the real technology is, what this little device is, and how it represents a capability to do some ‘heavy lifting’ in a deployed system.”
A little homework on the internet revealed that the little ‘eggshell’ had the advantage of a massive training set for its neural net based big brother, played on the internet by a huge number of people. Given the rules of 20 questions (Pick an OBJECT…Is it Animal, Vegetable, or Mineral?), this massive ‘honing’ of the taxonomy (folksonomy to the purists) results in a remarkably ‘prescient’ agent for playing (and winning) the game. This is the underlying technology that makes it tick. Further exploration gets interesting, and a dichotomy quickly appears: a) this clearly proves that AI can outperform a human in virtually every task or b) for categorization type tasks, with adequate training, this class of AI technique can do a remarkably good job (with a known empirically determined error rate). When caught up in the moment, you drift toward a); when considered in the light of day, b) makes you a better system developer.
The siren song comes from the technology being relatively invisible (exhibiting virtually no Observability in a Cognitive Systems Engineering sense), while being packaged in a compelling, emotionally appealing ‘toy’. During the discussion we came to characterize such toys as “things of small, controlled scale and scope”. As the discussion continued on and off for a few days, the distinction between toy and tool became a bit clearer. The scope limitation of the ‘toy’ that was essential to making it practical to implement and to make its performance ‘feel’ fast and accurate was exactly what made it a toy versus a tool. For example, it is impractical to expect the 20Q Toy to expand into the kind of tool needed to sort all of the “objects” referenced within the network traffic of a large corporation.
Unfortunately, the technology provider will blithely make such claims, point to the toy as evidence (often painted in the colors of the real world ‘tool’) and declare that “our technology will solve your problem.” A Cognitive Systems Engineer has to understand that underlying technology, understanding the affordances needed by a user in the real world, to do real cognitive work in the wild and look for the toys that really can grow to be tools.
//postscript// Sometimes, just sometimes, you can see past the Potemkin Village of the technologist’s toys. During an ARDA technical meeting, participants were presenting their kick-off thoughts…we began keeping score: 12 of the 13 vendors opened their presentations with “our technology solves your problem…Task 1 of our research is to understand your problem…” Note the oxymoron? Unfortunately we were to the only ones in the room that did…we were the 1 of 13.
If you haven’t seen it already, the trailer for Cage’s upcoming movie “Knowing” ends in a great line: “What do you do if you know Armageddon is coming and no one believes you?” The premise of the movie is the discovery of a prophetic piece of paper in a 50 year old time capsule that shows every major disaster…but ‘what happens when the numbers run out’? (The characters assume ‘the end’, it could be just the author ran out of paper to write on, or time in the elementary school class that authored it.)
The similarity to events that occur here several times a month is uncanny. Typically, one of the team will return from a meeting with the customer (user, prime contractor, program office) muttering, and pacing…in general showing a generous amount of frustration. The explanation normally centers around something like: “We can see that designing the automation that way will put the user in the ‘Surprise Mode Switch’ dilemma, and that will inevitably lead to an accident!”, or “They want to use their favorite design from a recent post on ManyEyes, but it has a False Emergence to it that will confuse users and will inevitably lead to…” Right, you’re sensing the theme. The good news: The list of these well proven, predictable system design errors is well known to good Cognitive Systems Engineers. For example, they form the core of Woods, Hollnagel 2006: Joint Cognitive Systems, Patterns in Cognitive Systems Engineering. Right, the other shoe…The bad news is that users, traditional systems designers, psuedo-designers, etc. etc. are *not* aware of these patterns, and refuse to accept them. Its a bit like not believing the earth is round, only much harder to prove to the unbelievers (of course that meme took quite some to to become a majority opinion as well).
It seems there are more than a few forces at work that make that the discipline frustrating:
a. traditional processes (e.g. DoD Acquisition Standards, CMMx initiatives, etc) and traditional training programs (e.g. Systems Engineering, Software Engineering, Design Programs, PMP, etc.) still teach ‘the world is flat’…and represent the dominant position among the decision makers/overseers. i.e. it takes an iconclast to accept Cognitive Systems Engineering into their long standing environment.
b. Each generation of system developer, not being exposed to the patterns of these “design disasters in waiting” is left to rediscover each and every one of them.
c. Users will move heaven and earth to succeed at their work, often despite the flaws designed into their tools. Woods calls this a “User Paradox”…so the system developers might never even see evidence of the flaw. (sometimes the traditionally trained accident investigators don’t even see it post hoc! they just arent trained to know what to look for.)
d. Ever changing technologies (e.g. each generation of programming language and environments) promises such great and wonderful things. Those promises distract the focus from what to do with them, and often become ends in themselves (rather than focusing on the characteristics of the tools being designed for users doing real work).
We used to say “Sometimes Cognitive Systems Engineering is like knowing tomorrow’s winning lottery number, but no one will believe you to buy the ticket.” Maybe now we’ll start saying “Being a Cognitive Systems Engineer is like being Nicolas Cage in ‘Knowing’.”
Stumble it!