Industry AnalysisIndustry Case StudyIndustry Insights & Trends



Explainable AI

Explainable AI (XAI) is an emerging field in machine learning that aims to address the lack of transparency of traditional AI algorithms and machine learning. As a trending technology in the AI space, it is seeing an increase in its adoption, driven by regulations and the need for understanding the rationale behind decisions reached by AI. These, including its untapped potentials, will continue to increase its adoption in the future. Made evidence in the number of startups and investment activities in the space, some leading startups include DarwinAI, Kyndi, Fiddler Labs, among others while these VCs include Intel Capital, Bloomberg Beta, etc. Driven by the need for XAI, large corporations like Intel and Wells Fargo have demonstrated their use of the technology.

Current Trends in the XAI Space

1. Implementation of the General Data Protection Regulation (GDPR)

  • Previously, financial regulators required financial institutions to use AI models that are explainable “to ensure that the models are robust enough to prevent bad lending decisions that may, in turn, jeopardize financial stability.”
  • This requirement focused only on data analysts, decision-makers, and the financial services market. However, with the GDPR coming on stream in 2018, it became the first regulation that extends ‘right to explanation‘ broadly — focusing mainly on the consumer.
  • “Article 13(2)(f) states that the controller (the entity who determines the purpose and means of processing) provide the data subject with further information necessary to ensure fair and transparent processing on the existence of automated decision-making, including profiling, referred to in Article 22(1) and (4) and, at least in those cases, meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject.”
  • This regulation has further driven the adoption of XAI and ethical AI for transparency and accountability.

2. More Guidelines Towards the Ethical Use of AI

  • In an attempt to implement an ethical use of AI, Nations and trade organizations are providing guidelines and regulations which call for the leverage of XAI. In 2019, the “EU presented ethics guidelines as it seeks to promote its own AI sector.” These frameworks were provided to boost trust in AI, and by extension, the implementation of XAI.
  • The seven key requirements of the guidelines include that AI be made transparent and accountable, which implies the use of XAI — as only XAI can ensure transparency, accountability, traceability, and auditability.
  • Later in that year, “Element AI, which was co-founded by Yoshua Bengio, one of the pioneers of deep learning, partnered with the Mozilla Foundation to create data trusts and push for the ethical use of AI.”

3. Increasing Adoption of XAI

  • The adoption of XAI is increasing, as more AI firms launch XAI platforms and enterprises employ deep learning to scale their AI capabilities.
  • The need to understand the rationale behind the decisions reached by AI technologies will continuously drive this trend. Moreover, the right to explanation empowers an individual “to receive the reasoning behind the output of an algorithm.”
  • This is commonly manifested among neural networks, wherein opacity is created by the inability of the network to explain outputs or results — a scenario usually known as AI black box.
  • To address these, large tech companies, such as Google, IBM, and Microsoft; as well as startups, such as Fiddler Labs and Kyndi are pursuing and pioneering XAI.
  • Furthermore, a report published by Tableau corroborates this trend, as the adoption of XAI is increasing because “organizations concerned with risks like financial services and pharmaceutical companies — are demanding data science teams to use models that are more explainable and offer documentation or an audit trail around how models are constructed.”
  • The growing number of other startups, such as Flowcast and DarwinAI; and investments by popular VCs, such as Intel Capital, Lockheed Martin Ventures, and Bloomberg Beta; in the XAI space testifies to the growth and high adoption of the novel technology.
  • Reports published by Fiddler Labs and TNW corroborate this trend, where there is an increasing number of XAI tools that are deployed in the tech space, emerging XAI startups, and the growing interest of VCs.

Future Trends in the XAI Space

1. Projected Growth/Adoption of XAI

  • Beyond 2020, analysts project that more and more companies, especially enterprises, will adopt and deploy XAI solutions. Gartner succinctly put that “by 2023, over 75% of large organizations will hire artificial intelligence specialists in behavior forensic, privacy and customer trust to reduce brand and reputation risk.”
  • Furthermore, Gartner projected that “30% of government and large enterprise contracts will require XAI solutions by 2025.”
  • Gartner posited the AI industry dive deeper into augmented analytics, including models and autogenerated insights, “the explainability of these insights and models will become critical to trust, to regulatory compliance, and to brand reputation management.”
  • Therefore, the benefits offered by XAI to address these challenges will be a major driver of this trend. These benefits include the increase of trustworthiness of AI outcomes, the reduction of risks pertaining to regulatory and reputational accountability including privacy violation risk, and the identification of bias and the potential sources.
  • In the banking sector, while XAI offers immediate solutions to risk and transparency challenges, a report published by Special Banking proves that the sector has only scratched a surface of what XAI can do. However, as banks continue to leverage the capabilities of the technology, it will increasingly become an integral part of their product development.

2. Causality from Observational Data: The Next Big Thing

  • While AI vendors have been integrating explainability features into their AI algorithms and systems to address questions like ‘why,’ finding out the causes would be preferred.
  • This can be achieved by conducting experiments, such as A/B tests, which are usually costly to create and manage and in some cases unfeasible.
  • However, a way around this would be to use the tons of observational data the AI has gathered. “It is an active field not only in academic research but also in startups.”
  • Some startups that are in the seed and Series A stages of establishing product-market fit include ClearBrain, Fiddler Labs, Kubit AI, and Sisu Data.

3. Future Regulatory Standards and Quality Assurance

  • With the need for transparent AIs or XAI, experts are calling for a method to assess AI model transparency. Last year, “the National Institute of Standards and Technology (NIST) convened a meeting on the advancement of AI standards as part of AI strategy plans from the White House.” The group deliberated on the concept of assessing models for transparency.
  • Elsewhere, “analysts from Cognilytica subsequently developed a multi-factor transparency assessment and contributed it to the Advanced Technology Academic Research Center (ATARC), a non-profit that seeks collaboration between government, academia and industry to address technology issues.”
  • These show that plans are being made and in the nearest future, there would be laid down industry standards and benchmarks on the assessability and rating of AI model transparency or XAI, by implication.

Leading Startups and the VCs Investing in XAI

1. Kyndi

  • Kyndi was founded in 2014, with its headquarters in California. It has 11-50 employees.
  • Kyndi is a provider of “leading artificial intelligence software that helps organizations answer their hardest questions. Kyndi can analyze long-form text and find actionable insights in a smarter, faster, and more explainable way than any other AI solution on the market.”
  • Kyndi was included as a leading startup in the XAI space for the following reasons:
    • AI Startups rated Kyndi as one of the top 27 startups developing AI for financial services, with the description as “an AI company to have built the first Explainable AI platform for critical government and commercial institutions.”
    • Venture Radar, a company/database that ranks the World’s innovative companies, also ranked Kyndi as one of the top Explainable AI companies.
    • CB Insights, in “its annual ranking of the 100 most promising AI startups in the world,” included Kyndi in its 2020 rankings.
    • In 2019, Gartner honored Kyndi as a ‘Cool Vendor‘ in enterprise AI governance and ethical response for their efforts in helping enterprises to better govern their AI solutions — making them more transparent and explainable.
    • Lastly, Kyndi has raised a total of $28.5 million in funding over 8 rounds, including their latest $20 million in series B funding in 2019.
  • Notable investors in Kyndi’s XAI technology include Intel Capital, PivotNorth Capital, UL Ventures, Citrix Startup Accelerator, among others.

2. Fiddler Labs

  • Founded in 2018, Fiddler Labs has 11-50 employees, with its headquarters in California.
  • Fiddler Labs provides XAI solutions for enterprises to “continuously monitor, explain, and analyze AI systems at scale. And with actionable insights, enable these enterprises to build trustworthy, fair, and responsible AI.”
  • Fiddler Labs was included as a leading startup in the XAI space for the following reasons:
    • “Fiddler Labs is a member of NVIDIA Inception, a program that enables companies working in AI and data science with fundamental tools, expertise and marketing support, and helps them get to market faster.”
    • In 2019, Gartner honored Fiddler Labs as a ‘Cool Vendor‘ in enterprise AI governance and ethical response for their efforts in helping enterprises to better govern their AI solutions — making them more transparent and explainable.
    • The World Economic Forum awarded Fiddler Labs as one of the 100 most promising technology pioneers of 2020 for their efforts in the use of its XAI solutions for business continuity and positive impact.
    • Venture Radar, a company/database that ranks the World’s innovative companies, also ranked Fiddler Labs as one of the top Explainable AI companies.
    • Lastly, Fiddler Labs has raised a total of $13.2 million in funding over 4 rounds, including the $10.2 million it raised in 2019.
  • According to the company’s website, notable investors in Fiddler Labs’ XAI technology include Lightspeed, Bloomberg Beta, Haystack, and Lux Capital. Others also include Amazon Alexa Fund and Lockheed Martin Ventures.

3. DarwinAI

  • Founded in 2017 and headquartered in Canada, DarwinAI has 11-50 employees.
  • Through its proprietary Explainable AI technology, the Gensynth Platform, DarwinAI enables deep learning developers to develop AI faster, deploy AI anywhere, and explain AI’s black box through actionable insights.
  • DarwinAI was included as a leading startup in the XAI space for the following reasons:
    • Venture Radar, a company/database that ranks the World’s innovative companies, also ranked DarwinAI as one of the top Explainable AI companies.
    • In 2019, Gartner honored DarwinAI as a ‘Cool Vendor‘ in enterprise AI governance and ethical response for their efforts in helping enterprises to better govern their AI solutions — making them more transparent and explainable.
    • “CB Insights also recently named DarwinAI a ‘high-momentum startup‘ in its ‘Game-Changing Startups 2020’ research report in the ‘AI transparency’ category, and also selected the startup for its AI 100, the company’s annual list of the 100 most promising private AI companies in the world.”
    • In Q1 2019, insideBIGDATA included DarwinAI in its list of the most important movers and shakers in the big data space for its relevance in the way they impact enterprises through leading-edge products and services — its XAI solution.
    • “DarwinAI was recognized as one of Ontario’s top Tech startups — as runner-up — in the hotly contest ‘Best Tech Startup’ category, a reflection of its growing status as one of the leading AI startups in Canada.”
    • TechTarget, an AI- and tech-focused media company, recognized DarwinAI as one of the AI vendors to watch out in 2020 and beyond, especially for its XAI solutions.
    • Lastly, “DarwinAI has raised a total of CA$3.9 million in funding over 4 rounds.
  • Notable investors in DarwinAI’s AI/XAI technology include Obvious Ventures, Inovia Capital, Garage Capital, Good News Ventures, and Shasta Ventures, among others.

4. Diveplane

  • Founded in 2017, with headquarters in North Carolina, Diveplane has 11-50 employees.
  • To support the ethical use of AI, Diveplane offers AI-powered business solutions across multiple industries that provide the full understanding and decision transparency to put machines and people in harmony by producing verifiable data intelligence.
  • Diveplane was included as a leading startup in the XAI space for the following reasons:
    • Venture Radar, a company/database that ranks the World’s innovative companies, also ranked Diveplane as one of the top Explainable AI startups.
    • TechTarget, an AI- and tech-focused media company, recognized Diveplane as one of the AI vendors to watch out in 2020 and beyond, especially for its XAI solutions.
    • In 2020, Gartner honored Diveplane as a ‘Cool Vendor‘ in the category of Core AI solutions. It declared that “data and analytics leaders can consider these Cool Vendors to address priorities around synthetic data generation, operationalizing and scaling AI projects, explainability and working with tough use cases.”
    • Lastly, Diveplane has raised a total of $6 million in funding, which was a seed round raised in 2018.
  • Notable investors include Presence Capital and Duke Angel Network. Others include popular celebrities and personalities, including Megan Rapinoe, Sue Bird, Meghan Klingenberg, among other individuals.

5. Z Advanced Computing, Inc. (ZAC)

  • ZAC was founded in 2013. It has its headquarters in Maryland, with 1-10 employees.
  • “ZAC is a disruptive 3D image recognition and search platform based on Explainable-AI. The platform mimics how humans discover, recognize, and learn by recognizing details beyond generic categories or classifications — a major limitation in neural networks.”
  • ZAC was included as a leading startup in the XAI space for the following reasons:
    • It is the first startup to demonstrate cognition-based Explainable-AI (XAI), which enabled it to be selected by the US Air Force, among the top teams to exhibit at a premier space industry event, “EngageSpace.”
    • The company already has an intellectual property portfolio comprising over 450 inventions, including 11 issued US patents.
    • “ZAC won the Judge’s Choice Award in Artificial Intelligence (AI) category at 2018 InnoStars Competition, attended by a few hundred leading investors and industry executives of major corporations from China.”
  • Notable investors include the US Air Force and following an earlier angel investment, the State of Maryland (TEDCO).

Examples of Large Enterprises Using XAI and Their Purpose

1. Intel

  • While AI algorithms, especially deep neural networks, are powerful based on predictive results, It has major drawbacks. These include build difficulties, run difficulties — often requiring significant resources for computations, and the lack of proper explanations or insights into decision-making processes.
  • “These limitations are highly problematic since if one doesn’t know how a model is reaching a decision, one can’t predict how and when it’ll fail.” Therefore, entrusting critical decision-making responsibilities to a system that cannot explain itself is counterproductive, precarious, and dangerous.
  • To solve this problem, XAI aims at imparting transparency into AI using methods that yield more explainable models without decreasing performance. For this, Intel employed DarwinAI’s Generative Synthesis solution that uses AI to build AI. Through this, the solution leverages machine learning to observe a neural network and based on the observations, it builds new and optimized versions of that network.
  • This reduces the difficulties and complexities of building new high-performance deep learning systems, generates highly optimized models or versions, and provides transparency into how and why a network makes decisions. Furthermore, with the complementing compatibility between DarwinAI’s Generative Synthesis solution and Intel Distribution of OpenVINO toolkit, the system is optimized for the easy deployment of deep learning applications to Intel architecture.
  • These resulted in the accelerated development and reduced time to market, faster training and reduced validation costs, and the optimization of neural networks.

2. Lockheed Martin

  • Lockheed Martin is one of the world’s largest aerospace, information security, and tech companies, with the majority of its revenues coming from defense contracts.
  • “Modern defense equipment is increasingly reliant on AI, as evidenced by the massive $800 million Department of Defense (DoD) AI contract recently awarded. Therefore, the need to properly understand the decisions of AI systems is likely to become a matter of literal life or death.”
  • Also, there is an increasing demand for more intelligent, autonomous, and symbiotic systems — “which XAI will be essential if future warfighters are to understand, appropriately trust, and effectively manage an emerging generation of artificially intelligent machine partners.”
  • Therefore, Lockheed Martin has strategically increased its efforts in this regard by partnering with DarwinAI to improve Lockheed Martin’s customers’ understanding of, and visibility into, its AI-based solutions — such as its autonomous and unmanned systems.
  • The aerospace giant has also partnered with Fiddler Labs to scale its XAI technology and then, apply it in the defense and aerospace industry. Through this, “enterprises gain complete visibility into their production AI systems to ensure high performance at all times.”

3. Wells Fargo

  • In heavily regulated industries, such as finance, “transparency, accountability, and trustworthiness of data-driven decision support systems based on AI and machine learning, or more traditional statistical or rule-based approaches, are serious regulatory mandates.”
  • Therefore, to keep up with these regulations while maintaining or increasing customer trust, Wells Fargo sought “to understand the challenges and opportunities with explainable AI.”
  • To achieve this, the multinational financial service company leveraged H2O’s XAI solutions to gain actionable insights and “to manage potential risks due to bias/fairness, conceptual soundness, implementation, and model change control.”


  • Other use cases of XAI can be found in Mixpanel’s Anomaly Explanations and Tableau’s Explain Data.

Use Cases of XAI in the Insurance Industry

1. Azur

  • Azur transformed the insurance industry by creating an augmented underwriting solution that enables real-time pricing and risk management.
  • Partnering with Temenos, Azur’s legacy systems were integrated with Temenos’ XAI platform. Through this, Azur helped brokers gain deeper insights into prospective clients. “Its users can also harness machine learning to rate the likelihood of conversion and propensity to claim as soon as a client requests a quote.”
  • With these capabilities, along with data-driven decision-making, cognitive bias is removed — “freeing underwriters to focus on more complex risks and broker relationships, and helping to tailor policies to individual risks for greater profitability.”

2. An Insuretech Company

  • An auto insurtech company partnered with PwC to speed up its claim estimation process through AI and analytics. While the insurtech company could leverage AI to scale the claim estimation process, the discovered the need to build an AI framework that can be trusted, both for estimators and customers.
  • To address this, PwC collaborated with the company’s data scientists to implement XAI. Through this, “the XAI solution could visually show why the model arrived at a particular prediction by making analytical determinations at every stage in the process.”
  • By doing this, “PwC greatly increased trust in the model as well and demonstrated its value to the data scientists, the estimators, the insurance companies and the insurance company’s clients.”
  • Other benefits include reduced rework and improved customer experience through reduced cycle times.

Research Strategy

The research team was able to provide three examples of large enterprises that are using XAI and the purpose behind the use; three current and future trends each, including trends around regulation, legislation, and adoption of XAI; and five leading startups in the XAI space and the VCs that are investing in the technology. For the trends, the research relied on multiple industry and/expert reports to synthesize the trends based on those that are widely discussed, evidenced by available corroborating sources. The leading startups were determined based on various recognition, awards, and funding each startup has received.
However, after scoring industry sources, expert analysis, white papers, and press reports, the research team could not determine any use case of large insurance companies that are using XAI. Further research into companies that offer XAI solutions was conducted to see if their customers and customer stories included insurance companies, including large insurance companies. Through this, we could only identify Azur, an insurtech company and an unnamed insurtech company, as provided by PwC. All other relevant use cases found were for companies that provide software solutions for the insurance industry, including how their solutions can be leveraged by the industry. Hence, these use cases were provided as alternatives.

Glenn is the Lead Operations Research Analyst at Simple Manifestation with experience in research, statistical data analysis and interview techniques. A holder of degree in Economics. A true specialist in quantitative and qualitative research.


Previous article


Next article

You may also like


Leave a reply

Your email address will not be published.