The Myth of Neutrality

Why the Next Generation of Human and Artificial Intelligence Must Be Designed for Correctability

8/26/202614 min read

a wooden judge's hammer sitting on top of a table

Author: Trang Phan

Introduction — We Are Building the Most Measured Civilization in History

We are building the most measured civilization in history. Companies can see clicks, purchases, abandonment, retention, location, response time, productivity, sentiment, risk scores and behavioral patterns at extraordinary scale. Artificial intelligence can process quantities of data that no human research team could inspect manually. Governments, institutions and businesses increasingly make consequential decisions through dashboards, models and algorithms.

We have never had more visibility. And yet there is a strange possibility hiding inside this abundance of data: we may be becoming more precise about a progressively smaller representation of reality.

That distinction matters. A system can measure something with extraordinary accuracy and still be measuring the wrong thing. A model can predict behavior extremely well and still misunderstand what the behavior means. A dashboard can report every metric correctly and still hide the phenomenon that ultimately destroys the business. A user can complete a task successfully while losing trust. An employee can become more productive while becoming less capable of sustaining the work. An AI can produce a correct answer while gradually reducing the human's ability to recognize when the answer is wrong.

The central problem is not bad data. It is deeper. Before anything can be measured, something must first be made observable. Before it can become observable, someone—or some system—must distinguish it from everything else. And the moment that happens, a boundary has already been drawn around reality. That boundary is the beginning of the observer problem.

For decades, human-centered design has largely asked: how do we understand the user better? The AI era forces a second question: how does the system through which we understand the user determine what we are capable of seeing? That is a different level of design. And it leads to a different standard for intelligence. The goal should not be neutrality. The goal should be correctability.

1. Every Observer Has a Field of View

Consider something as ordinary as a camera. The camera can produce a perfectly accurate image of what falls within its frame. But the accuracy of the image tells us nothing about what lies outside the frame. The same principle applies to every observation system. A microphone has a frequency range. A survey has questions. A usability study has tasks. A dataset has a schema. An analytics platform has events. A machine-learning model has inputs. A researcher has attention. A company has KPIs. A regulator has reporting requirements. Every system that observes the world has some form of visibility envelope.

This is not necessarily a defect. It is a condition of observation. Human cognition itself works through selection. Research in cognitive neuroscience consistently describes attention as a limited-capacity process that selectively prioritizes information relevant to current goals. We do not process everything and then decide what matters; selection is part of the mechanism that makes useful cognition possible.

The implication is more important than it first appears. There is no perception without selection. No research without selection. No measurement without selection. No dataset without selection. No AI inference without a prior representation over which the inference operates. The problem is therefore not simply that humans are biased and machines might become less biased. Machines inherit another observation architecture. Their categories, data, objectives, training distributions, labels, context windows, sensors and evaluation procedures determine what becomes visible to them. The machine changes the observer. It does not remove the observer.

2. Measurement Begins Before the Number

Imagine a product team measuring onboarding completion time. The resulting number might be extremely precise. But considerable design occurred before the stopwatch started. Someone decided what counts as the beginning of onboarding. Someone decided what counts as completion. Someone chose which participants belong in the sample. Someone determined which interruptions count. Someone decided that speed matters. Someone decided what constitutes an error. Someone decided whether hesitation is friction, confusion, reflection or something else. The arithmetic may be objective once those decisions have been made. But the measurement architecture was designed.

This is not a fringe philosophical objection. Measurement science has long recognized that validity concerns the interpretation being made from measurements, not merely the reliability of the instrument producing them. Construct validation requires a theory connecting the thing we care about to the thing we actually observe. Validation is therefore an ongoing process rather than a permanent property bestowed on a metric.

That distinction exposes a mistake modern organizations make constantly: precision is confused with completeness. A number can become more precise without the organization becoming more correct. A million observations of the wrong proxy do not produce the missing variable. A larger dataset does not automatically enlarge the field of view. More measurement does not necessarily mean more reality. Sometimes it simply means greater confidence inside the same observational boundary.

3. Dashboards Do Not Merely Describe Organizations. They Teach Organizations What Reality Is.

This is where the observer problem becomes institutional. Suppose an executive dashboard contains four prominent numbers: revenue, conversion, retention, engagement. Those metrics are not merely reporting the organization. They are gradually defining the organization. Teams receive targets based on them. Capital is allocated around them. Experiments are evaluated through them. People are promoted for improving them. Products evolve toward them. Eventually, thousands of local decisions begin converging on the ontology embedded in four boxes on a screen.

Now imagine what the dashboard does not contain: regret, cognitive burden, dependency, loss of agency, trust degradation, long-term skill erosion, unintended exclusion, or harms occurring outside the measurement horizon. Their absence does not mean they are absent from reality. It means they are absent from the organization's machine-readable reality.

That distinction becomes increasingly consequential once optimization enters the loop. Measurement becomes target. Target becomes incentive. Incentive reshapes behavior. Behavior creates new data. New data reinforces the measurement system. What started as observation becomes governance. This is why Goodhart-type effects matter so much: metrics can change meaning when they become targets of optimization. The dashboard is therefore not simply a mirror. It is an intervention. And increasingly, the same is true of AI.

4. AI Industrializes the Observer Problem

The popular story says that AI allows us to move from subjective human judgment toward data-driven decision-making. There is truth in that. Algorithms can discover patterns humans miss. They can apply criteria more consistently. They can analyze enormous datasets. They can reduce some forms of arbitrary human variation. But none of this eliminates the prior problem of representation.

Before an AI system can learn, the world has already been transformed. Reality becomes events. Events become records. Records become datasets. Datasets become selected variables. Variables become features, tokens or representations. Representations become model inputs. Outputs are evaluated using benchmarks. Benchmarks are interpreted by people. Then people decide which outputs matter. The supposedly objective machine sits inside a long chain of human and technical distinctions.

This is why some of the most influential work in responsible machine learning has focused not only on model architecture but on documenting where datasets came from, why they were created, what they contain and where they should not be used. Datasheets for Datasets was built around precisely this need for visibility into dataset motivation, composition and collection. Model Cards similarly argued that model performance should be reported with intended use, evaluation context and limitations rather than reduced to a single score. These developments point toward a much larger principle: the epistemic boundary of an AI system is part of the system. It cannot remain documentation added after deployment. It is architecture.

5. The Most Dangerous Data State Is Not Zero. It Is Unknown Mistaken for Zero.

Suppose a team asks whether a new financial application increases customer anxiety. No anxiety measure was collected. No interview question addressed it. No physiological signals exist. No observational protocol included distress. What is the answer? Not low anxiety. Not acceptable anxiety. Not zero anxiety. The correct answer is: unknown.

That seems obvious when stated explicitly. Organizations violate it every day. What is not represented in the information system gradually becomes indistinguishable from what does not exist. This is one of the deepest risks of data-driven management. We collapse very different epistemic states: observed absence, unobserved, unmeasurable with the present instrument, insufficient sample, unknown, and genuinely absent. These states should never mean the same thing. Yet once they enter organizational systems, absence often becomes zero because zero is easier to compute.

AI can compound this error. Models are exceptionally good at filling representational space. When the input is incomplete, a sufficiently capable generative system may still produce a fluent output. Fluency can conceal the difference between evidence and interpolation. Confidence can therefore increase precisely where epistemic humility is required. The proper response is not merely better accuracy. It is architecture that preserves the state unknown. That may turn out to be one of the most important capabilities of trustworthy intelligence.

6. Human-Centered Design Has Focused on the User. It Now Needs to Examine the Observer.

Human-centered design already recognizes the importance of context, iteration and understanding users. ISO 9241-210 formalizes human-centered design as a lifecycle process for interactive systems rather than a one-time interface exercise. But the AI era exposes a missing second-order discipline. Designers need to study not only the human being observed, but the system performing the observation.

Traditional design logic might look like observe, define, design, test. The deeper loop is position, distinguish, observe, represent, measure, interpret, define, design, intervene, observe again. The observer exists before the research finding. That matters because designers do not discover a completely formed problem waiting in reality. They participate in constructing its boundary.

Consider the statement: "Users are abandoning onboarding." That may be an observation. But the moment it becomes "we have a conversion problem," a theoretical choice has been made. Perhaps abandonment reflects confusing UX. But perhaps the product requests too much personal information. Perhaps customers correctly realize the service is unsuitable. Perhaps a legal disclosure changes their assessment of risk. Perhaps a moment of friction gives them enough time to reconsider a financial decision. Perhaps low trust—not usability—is the governing variable. Each framing produces a different intervention. If the problem is defined as friction, the system removes friction. If friction was protecting reflection or informed consent, the optimization succeeds at its metric while degrading the human system. That is not primarily an interface failure. It is an observer failure.

7. The Next Frontier of Design Is Negative-Space Intelligence

Design disciplines have become extraordinarily sophisticated at analyzing what people do. The next frontier is understanding what the observation system could not see. What was never asked? What was excluded by the sample? What could not be captured by the instrument? What appeared after the study ended? What behavior was suppressed by the experimental environment? Which population never entered the dataset? Which consequence happened outside the business's reporting boundary? Which variable disappeared when qualitative experience became a dashboard metric? Which human capability deteriorated while task performance improved?

This does not mean measuring everything. That is impossible. It means representing the unmeasured explicitly. A serious research report should not contain only findings, insights, and recommendations. It should also contain known unknowns, unobserved variables, alternative explanations, measurement boundaries, time-horizon limits, and conditions under which the interpretation should be revised. That is not methodological weakness. It is epistemic strength.

8. AI Makes This Urgent Because Machine Observations Become Actions

A weak human research model can produce a bad presentation. A weak machine representation can produce millions of automated decisions. Scale changes the consequence. An AI system's representation affects what applicants appear risky, which patients receive attention, what content people see, which employees are flagged, which transactions are challenged, which customers receive offers, which information becomes retrievable, and which possibilities an autonomous agent considers.

This is why modern AI governance increasingly treats evaluation as a lifecycle activity rather than a single benchmark score. NIST's AI Risk Management Framework organizes risk work around Govern, Map, Measure and Manage, explicitly treating context, measurement and ongoing management as connected functions. Current NIST work continues pushing toward test, evaluation, verification and validation of AI systems in their actual application context rather than assuming a generic benchmark establishes trustworthiness. That evolution tells us something important. The problem is moving from "is the model accurate?" toward "is the system's representation adequate for the decision we are allowing it to influence?" That is a much harder question. It is also the right one.

9. More Human Oversight Does Not Automatically Solve the Problem

There is an appealing solution to all of this: keep a human in the loop. But humans are observers too. A person asked to review an AI recommendation may not possess the time, information, authority or understanding required to challenge it. Human oversight can become ceremonial. The machine produces the answer. The human presses approve. Responsibility remains nominally human while epistemic control has quietly moved elsewhere. Recent work on meaningful human oversight makes this distinction explicit: the existence of a human reviewer is not sufficient; the reviewer needs the capacity to understand, evaluate, intervene and override.

That observation leads to a deeper principle. Correctability is a property of the relationship between observer, system and authority. It is not enough for the AI to be technically modifiable. Someone must be able to identify when modification is needed. Someone must possess evidence capable of challenging the system. Someone must possess authority to intervene. And the surrounding institution must tolerate the contradiction. Otherwise "human control" is only an organizational story.

10. Correctability Is Stronger Than Neutrality

This is where the argument changes. If every bounded observer must select, then eliminating every observational bias is impossible. But that does not lead to relativism. It leads to a stronger standard. A good observer can state its position. A good measurement system can expose its boundary. A good model can reveal its intended scope. A good research process can preserve competing explanations. A good institution can distinguish absence from unknown. A good AI system can expose uncertainty. A good governance structure allows outputs to be challenged. A good decision process preserves evidence capable of overturning the decision.

In other words: the objective is not an observer with no perspective. It is an observer whose perspective can be inspected and corrected. This idea already appears in different forms across measurement science and AI governance. Construct validity is treated as an accumulating process of testing interpretations rather than something permanently "proven." AI governance increasingly emphasizes documentation, evaluation, contestability, oversight and mechanisms for correction. Research on corrigibility asks whether AI systems remain amenable to modification or shutdown rather than resisting human intervention. These fields are converging on the same architecture from different directions. And that architecture deserves a name: correctable intelligence.

11. Correctable Intelligence Requires More Than Being Able to Change the Answer

Correction is often confused with update. They are not the same. A system can change constantly without becoming more correct. Social media changes constantly. Markets change constantly. Models are retrained. Organizations reorganize. Individuals revise opinions. None of this guarantees improvement.

Correction requires a relationship between a representation, evidence capable of challenging it, a mechanism for detecting mismatch, authority to change the representation, and memory of why the change occurred. Without evidence, change is drift. Without contradiction detection, change is adaptation without direction. Without authority, the correction cannot occur. Without provenance, the system cannot distinguish learning from arbitrary revision. Without memory, the same failure can return. So a correctable system is not merely flexible. It is structured for self-revision without losing epistemic continuity.

That distinction matters enormously for AI. An endlessly adaptive system is not necessarily safer than a fixed one. If it adapts without preserving the evidence, constraints and reasons behind prior decisions, it may become harder to govern rather than easier. Correctability therefore requires memory and boundaries as much as adaptability.

12. The Real AI Divide May Be Between Systems That Optimize and Systems That Can Be Corrected

The first generation of machine intelligence has been dominated by optimization. Maximize prediction accuracy. Minimize loss. Increase engagement. Reduce cost. Improve completion. Raise reward. This paradigm is enormously powerful. But optimization begins after the objective has been represented. It does not answer whether the representation itself is adequate. That becomes dangerous when systems operate in human environments.

A model may perfectly optimize an incomplete objective. A recommendation system can optimize engagement while eroding user agency. A workplace AI can optimize measured productivity while degrading unmeasured expertise. A customer-service system can reduce average handling time while destroying trust. A design tool can increase output while narrowing creative diversity. A learning assistant can improve immediate task performance while weakening independent problem-solving. Every one of these systems could be "working." The question is: working according to whose representation of success? The next generation of AI needs to do more than optimize inside a frame. It needs systems and institutions capable of revisiting the frame.

13. This Changes What Leaders Should Ask

Executives increasingly ask: what can we automate? What can AI predict? How accurate is the model? How much productivity can we gain? Those are reasonable questions. But they are downstream questions. The upstream questions are more consequential: what reality did we make visible to the system? Which variables did we exclude? What does the model have no way to know? Which metric becomes dangerous if optimized too aggressively? What happens outside our measurement horizon? Which human capability could decline while our performance metric improves? Who can challenge the model's representation? What evidence would cause us to change the objective itself? Can the system preserve "unknown," or must it always produce an answer? Can we reconstruct why the organization came to believe what it believes? These are no longer research-team questions. They are governance questions. And increasingly, they are strategy questions.

14. The Designer of AI Is Designing a Way of Seeing

The design profession is changing accordingly. Designers once primarily shaped artifacts. Then experiences. Then services. Then systems. AI pushes designers one layer deeper. They are increasingly shaping what the machine observes, which context it receives, what becomes memory, what gets forgotten, when uncertainty is shown, how recommendations are framed, which decisions remain human, when intervention is allowed, which outcomes are measured, and how feedback changes the system. That means the designer is helping construct the epistemic architecture through which the organization sees its world.

This is a much larger responsibility than interface design. The designer is deciding what can become visible. And because what becomes visible can become measurable, and what becomes measurable can become optimizable, those decisions eventually shape behavior. The design of observation precedes the design of action.

15. Human-Centered Design Should Become Observer-Aware Design

Human-centered design does not need to be discarded. It needs a second loop. The first loop studies the person interacting with the system. The second loop studies the architecture through which the first loop produces knowledge. That means asking, continuously: what did we distinguish? Why? What did we measure? Why? What did the method exclude? What assumptions connect the measurement to the conclusion? Which alternative explanation remains plausible? What consequence occurs at another scale? What changes over a longer horizon? What would make us revise the model?

The resulting process is not empathize, define, ideate, prototype, test alone. It becomes something closer to: position, distinguish, observe, represent, measure, expose unknowns, interpret, design, intervene, measure consequences, challenge representation, correct. The last step is the crucial one. Research should not only change the product. Research should be capable of changing the observer.

16. What Is Not Measured Still Exists

This is the principle organizations most need to recover. Reality does not begin when it enters a database. Human experience does not begin when it becomes an event property. Risk does not begin when it becomes a dashboard indicator. Trust does not begin when a survey measures it. Harm does not begin when a reporting system recognizes it. And intelligence does not become objective simply because its judgment arrives as a probability.

All intelligent systems operate through representations. Human beings do. Institutions do. Scientific disciplines do. AI systems do. The mature question is therefore not whether the representation is perfectly neutral. It cannot contain everything. The question is whether the system knows enough about the boundary of its representation to remain capable of correction. That is the real dividing line. An intelligent system that cannot revise how it sees can become trapped by its own sophistication. A human can become trapped by a worldview. An organization by its KPIs. A scientific field by its categories. An AI by its objective function or accumulated model. Greater intelligence inside a closed representation does not necessarily solve the problem. It may simply optimize the error.

Conclusion — The Future Belongs to Correctable Systems

We are entering an era in which machines will increasingly participate in observation itself. They will summarize. Classify. Recommend. Prioritize. Evaluate. Predict. Remember. Act. As their capability grows, arguing about whether humans or machines are more objective will become less useful. Both are bounded observers. Both operate through representations. Both can miss what their architecture does not expose.

The more important distinction will be between systems that can discover and correct the limits of their representations—and systems that cannot. That changes the ambition. The strongest organization will not be the one with the most data. It will be the one most capable of discovering what its data does not contain. The strongest research team will not be the one that claims the greatest certainty. It will be the one that knows exactly where certainty ends. The strongest AI will not merely generate the best answer. It will operate inside an architecture where the answer can be challenged, traced, invalidated and repaired.

The strongest designer will not merely understand users. They will understand the machinery through which users become knowable to the organization. And the strongest intelligence—human, institutional or artificial—will not be the intelligence that never gets the world wrong. No such intelligence exists. It will be the intelligence that can discover when its model of the world has failed, preserve the evidence of that failure, and change without pretending that the previous model was never there.

That is a more demanding standard than neutrality. It is also a more useful one. The future of intelligence is not objectivity without an observer. It is observation with the capacity to correct itself.