The talk of the town this year, last year, and every single second since the Nov 2022 release of ChatGPT has been the use of applied and generative AI. Artificial intelligence has been a concept in data for decades and has taken on highly publicized initiatives like Watson, but never before has use of the technology seemed so pervasive in our society. It is no longer just those within the data sphere discussing pros or cons, debating over hallucinations or false feedback, or getting into the intricacies of AI ethics. AI has officially become dinner table talk. The deployment of products like Claude, Microsoft’s Gemini, or Databricks’ Genie have shown that AI can be engrained in business and personal use alike and is improving individual contributor efficiency across the board while also allowing more effective and empathetic management from higher level stakeholders.
Beyond generative AI lies other applied AI capabilities. Models are moving away from raw R&D into directed use cases that are impacting how and why decisions are made. NLP is increasing the efficiency and speed of migration from raw notes or PDFs to structured elements of data that can be leveraged by users. And agentic AI allows those models to work together with functions and other applications to carry out tasks that require little to no human intervention and enable those resources to focus on higher level tasks.
Adoption of AI throughout enterprise has come quickly and, unfortunately, is experiencing growing pains as more users are expected to utilize the technology. This may involve anything from entering proprietary information like client details into a public model to deploying custom models before testing for bias. In some instances this can be comical, like when a Chevy dealership’s chatbot would suggest customers purchase a Ford F-150[1] or when major publications put out summer reading lists filled with books that didn’t exist.
Other times this can be detrimental to both individuals and organizations. For example, in 2024 Grok posted on X accusing Klay Thompson (at the time playing for the Golden State Warriors and now with the Dallas Mavericks) of shattering Sacramento home windows with bricks. In another instance, the iTutor Group settled for $365,000 after using an AI tool to scan recruits. The tool automatically rejected female applicants aged 55+ and male applicants 60+ and ultimately rejected over 200 qualified applications automatically because of the model error.[2]
AI has the potential to be a great boon to how we live our lives. However, these solutions need to be well governed to build trust and create accountability while also ensuring quality models that do their jobs as expected. Bias, model drift, poor production deployment, and inadequate testing are just some of the elements of problematic models that cause significant harm to individuals, reduce trust in the technology, and cost businesses significant sums in redevelopment and litigation. Processes like automated testing, defined organizational AI ethics, comprehensive CICD, data quality minimum requirements, version control, drift monitoring and mitigation, and many others need to become a baseline expectation at any organization that intends to deploy AI in a responsible way. Not only that, but governing bodies and RACI networks need to be established to ensure these protocols are defined, maintained, and enforced. Established AI governance should be any organization’s first step towards AI development, and if governance has not been deployed any AI development needs to be paused while requirements are being decided upon.
At the root of AI governance is the desire to provide revolutionary technology that is high quality and drives society as a whole forward, not just a privileged few. Because of this, organizations need to follow basic principles when developing their AI programs. The extent to which a company adopts these principles or which governing elements are derived from these principles will vary based on the organization’s needs, ethics, and goals. However, to develop a conscientious program with scalability and longevity they need to be implemented in some way or another.
Models should regularly be tested to reduce bias and align with company ethics. This means AI products should promote unbiased decision-making and equitable treatment for all demographics. Additionally, processes need to be designed to systematically test models for fairness both before deployment and on a regular cadence in production. Again, AI should not be benefiting just the privileged few.
Models should ensure human agency is available with boundaries by establishing clear roles and responsibilities as well as human supervision mechanisms. This also means defining times at which human intervention can occur and when it cannot. For example, there are times at which an individual may need to override their autonomous vehicle, however they should be prevented from doing that while intoxicated.
Developers should eliminate exposure of sensitive data within the model and ensure where that data is required information is secure. Data privacy needs to be prioritized and safeguarded which includes minimizing the use of sensitive data across all models. The models themselves also need to be secured, especially in the case of AI agents. Automation built into those agents should be monitored and regularly audited to ensure long-term security.
Models should be well documented and replicable. Organizations need to ensure AI systems provide clear, explainable, and understandable decisions and prioritize clear box models as opposed to black box models. Any AI product should be easily replicable by team members to validate the solution itself and audit logic trails.
Models should be built with consideration of future use, growing data sets, and breaks in continuity. Build reliable, innovative and secure AI that can be functionally deployed with high performance and scalable architecture. Additionally, be sure to audit and maintain data quality of the underlying data sets on a regular basis. Data scientists and owners alike should also understand and mitigate excessive energy consumption with efficient model construction to reduce cost and resource needs.
Processes should ensure models are moved to production in an efficient, effective, and comprehensive method. Implement change management as outlined as a key element of data productization to ensure the data team understands and properly deploys new processes, and define CICD processes that include automated migration from lower to higher environments, automated testing, and comprehensive version control. Finally, apply logging and observability features to provide model operations insight.
Ethics, bias, and maintenance of expected use are concepts of data and software development that have existed in the background for decades. With the expanded use of AI however these elements take a front seat, requiring significant thought, assessment, and scrutiny. AI has the potential to change people’s lives whether that be through job application analysis, college acceptance analysis, recidivism likelihood decision, patient care recommendations, and more. With that in mind, these features of AI governance need to be at the top of mind for anyone even considering deploying AI solutions. It is difficult to walk back a mistake that can detrimentally impact an individual or a group. Addressing these concerns before deploying AI solutions will not ensure that everything will go perfectly as you deploy those solutions, or that mistakes won’t be made by the technology. What is will ensure is that those mistakes are few and far between, and when they do happen they are detected before impact is seen instead of having to be reactionary after the fact. Those organizations that act on AI governance before acting on the solutions will ultimately be the leaders in the space with scalable, efficient, and effective AI solutions; those that don’t will encounter error after error as they continue to deploy ungoverned AI solutions to production and will spend a large percentage of their day putting out fires instead of building innovative products.
Amanda Darcangelo is a Lead Data & Analytics Consultant at CTIData.
[1] https://www.freep.com/story/money/cars/general-motors/2023/12/19/chevy-dealership-ai-chat-ford-f-150/71960060007/
[2] Examples from this and the previous paragraph were identified through https://www.cio.com/article/190888/5-famous-analytics-and-ai-disasters.html
© Corporate Technologies, Inc.