ShadowBox Winter Newsletter 2021

As a new year begins, we at ShadowBox continue to be impressed by the importance of expertise. The skill and adaptability shown by so many in the face of the pandemic underscores the significance of expertise, and why, more than ever, organizations need to promote and develop expertise in their workforce. Our training courses are designed to accelerate expertise by addressing the cognitive dimension. This year, we made significant strides with our three main training domains: law enforcement, child welfare, and process control in the petrochemical industry. We also made exciting developments in new domains (such as healthcare and human interaction with Artificial Intelligence systems), and we are expanding our client base. This newsletter describes how we’ve expanded our knowledge and our own expertise.

Project Innovations

UTA College of Nursing

While small in scope, our project with the University of Texas at Arlington’s College of Nursing and Health Innovation may turn out to be huge in terms of impact. We had just enough funding to develop one video-based ShadowBox training exercise for a cardiac care situation. Just one scenario. Working with the dedicated staff at UTA, we identified the cognitive skills we wanted to exercise and the critical cues we wanted reflected in the scenario. We developed a script, from which UTA produced a training video.

But the scenario was only about a minute and a half long. How much can you get out of 90 seconds? A lot, as it turns out. We were astonished by the things that the skilled nurses on the staff picked up. We prepared the Cue Detect version of the scenario and, with some nervousness, turned it loose on 190 nursing students. We’d never trialed the software system with such a large group before, and a remote one at that.

We shouldn’t have worried. The system worked without any glitches, and the nursing students went through the training with no problems. Our sponsors at UTA think that this demonstration could be the start of a different approach to nursing education. Instead of relying so heavily on readings and lectures, now they can have the students observe a tough case, enter their reactions, and get immediate feedback on what the experts were picking up — cues that they had missed. The nursing students were truly seeing this situation through the eyes of experts.

The Guardian Mindset: De-escalation and Decision Making for Law Enforcement

Led by John Schmitt (Senior Research Associate), we are making final preparations to kick off a decision skills course for police officers. The course is undergoing final certification by the California Commission on Peace Officer Standards and Training (POST). It will be made available for online and in-person delivery to California law enforcement officers starting in February 2021. The course is based on 16 challenging scenarios (developed from real-life incidents) and is delivered over 24 instruction hours.

We started developing the course in 2015 through a collaboration with California POST. Our goal was to develop a set of learning tools to address the perceived lack of critical thinking in initial response incidents. ShadowBox conducted cognitive task analysis (CTA) interviews and observed officer decision making. We identified a core set of 11 cognitive skills and strategies, such as sensemaking, prioritizing, and anticipating. Our ShadowBox scenarios were designed to improve officers’ skills in these areas.

Workshop on Recognizing and Preventing Fixation Errors

Gary Klein (President and CEO), Joseph Borders (Research Associate), and Adam Zaremsky (Research Assistant), in collaboration with project champion Ron Besuijen (Technical Training Specialist at NOVA Chemicals), led a workshop for process control operators and trainers in the oil, gas, and petrochemical community on December 9th and 10th. The virtual workshop 1) presented causal factors that lead to fixation and diagnostic errors, 2) increased participant knowledge for how to recognize fixation, and 3) described tactics and suggestions for prevention and correction. Both workshops produced rich discussion and generated new ideas which we are eager to cultivate and refine for our project sponsor, the Center for Operator Performance (COP). The workshop participants gave the workshop very high evaluations. When asked if they thought any new trainer would benefit from the workshop, all but one of the respondents indicated that they strongly agreed, 5 on a 5-point scale, and that one indicated that they agreed, 4 on a 5-point scale. A video of the major lessons and tactics from the workshop will be available to view for affiliates of the COP in February 2021.

Cognitive Support for Assessing XAI Systems

Our team, led by Joseph Borders, has continued to develop tools to enhance the effectiveness Artificial Intelligence and Machine Learning technologies. Our latest development, sponsored by the Air Force Research Laboratory, includes developing cognitive interview guides to assess usability and explainability of the AFRL COVID-19 dashboard. We have also designed a discussion guide for technology developers. These discussions will facilitate the creation and validation of a Cognitive Tutorial. A Cognitive Tutorial provides developers and users with a better mental model of a system (how it works, how it fails, why it fails, and how to adapt).


Gary’s team, working with DARPA’s XAI program, continues to make strides in the development of an Artificial Intelligence Quotient (AIQ) toolkit. These tools will improve users’ mental models, increase user satisfaction, and increase explainability of AI systems. Work on one of the tools, the Discovery Platform, has recently been completed. The Discovery Platform is a set of representation tactics for discovering and diagnosing an AI system’s strengths and limitations. In the coming year, Gary and his team will be generating more effective ways for the DARPA community to conduct research under a new approach Gary is calling Minimum Necessary Rigor. MNR is a way to frame research to reduce or remove excessive rigor, expense, and time, under the appropriate conditions.

Child Welfare

Led by Emily Newsome (Research Associate and Assistant Manager) ShadowBox continues to be utilized by the largest child welfare agency in Ohio for continuing worker education and for new worker training. We are expanding to other regions, including rural counties in Northwest Ohio. We are working with counties to implement a training program that consists of 23 challenging scenarios illustrate common child welfare themes. These scenarios have been shown to improve the critical thinking involved in making complex decisions about child welfare. If your organization is interested in scenario-based training designed to bring new staff up to speed, and further train existing staff, please contact us at

Announcements and Developments

Welcome Reza!

We are excited to announce the addition of a Postdoctoral Research Associate to our team. Mohammadreza (Reza) Jalaeian, recently completed his Ph.D. in Learning Systems Design & Technology from Southern Illinois University Carbondale. His dissertation is titled: Applying Video-Occlusion Research Method to Guide the Design of a Perceptual–Cognitive Training Program for Law Enforcement Officers. We are looking forward to applying his expertise with police officer training and designing learning systems. He will be a great addition to our team.

New Book

Gary is writing a new book for MIT Press publishing company. The book will be a collection of essays he has written on the topic of Cognition. While cognitive processes have been studied for over a century, this book will aim to change how readers think about cognition in real, messy, complex environments, not the neat, controlled laboratories that are so often used. The ideas that will be presented are cultivated from Gary’s 35 years of work understanding cognition through direct observation, and in collaboration with the Naturalistic Decision Making community.


In collaboration with other members of the NDM community, ShadowBox is helping plan and prepare for the next NDM conference, hosted in collaboration with the Resilience Engineering Association (REA) this upcoming June 21-25, 2021. We are exploring ways that these two communities can intersect, exchange ideas, and further each other’s pursuits. This past November, Gary co-led a webinar with Sudeep Hegde (Assistant Professor of Industrial Engineering at Clemson University), about NDM and RE tools in workplaces to get jobs done, as well as where these two communities tools converge and diverge.

Upgrading the Web-based ShadowBox Application

Progress continues rapidly on our upgrades to the web-based ShadowBox scenario training application. In addition to greater user friendliness, more intuitive displays, and improvements to the facilitation mode, some of the coming features we’re looking forward to include:

• Scenario of the month: This feature will allow visitors to our website to access a free ShadowBox scenario exercise each month.

• Cue Detect improvements: Our patented Cue Detect feature allows a user to identify visual and auditory cues during video training. After watching and identifying cues, users get to see what the experts’ noticed, and the rationale behind the experts’ choices.

• Dual perspective decision points: A ShadowBox exercise typically has someone responding to different questions, such as, “What are your priorities in this moment?” We are creating a new question feature that allows one to respond in two ways: 1) from their own perspective, and 2) from the expert perspective. The idea is to let someone attempt to predict how the experts would respond and reflect on whether these responses are different from their own.

New Website

We are partnering with Tony Leone Design LLC to develop a brand-new ShadowBox website. We are excited to reflect our company’s innovative strategies with a new, modern design. We plan to implement the new website by February 2021.

Featured Posts
Recent Posts
Search By Tags
No tags yet.
Inside the Mind of 
Dr. Gary Klein
  • Twitter Basic Square

© 2015 by ShadowBox. All rights reserved.

Designed by ORA Web Design  937.723.8038

  • Twitter Social Icon