The 24-Hour Compliance Test
What regulators actually expect to see
Imagine a regulator contacting your organization on a Tuesday morning with a specific request : produce evidence of EU AI Act compliance for one of your high-risk AI systems within 24 hours. The request is named; a particular system, a particular obligation surface, a particular set of articles. The deadline is short. The penalty for non-production is not theoretical.
Many organizations won’t probably pass this test. The reason is rarely what they expect.
The gap is not in the underlying compliance work. It is in documentation organization and retrieval readiness. The work has been done : risk assessments completed, logs collected, training records maintained, conformity declarations signed. What is missing is the structure that allows those artifacts to be located, verified, and presented under time pressure. When the request arrives, the work that took weeks to produce must be retrievable in hours; and many organizations have not built the systems that make that possible.
This is the eighth article in our ten-part series on AI governance. The previous article examined where organizations actually stand on maturity. This one examines what regulators expect to see when they ask; and the practical difference between being compliant and being able to demonstrate it.
The 24-Hour Test as a Practical Diagnostic
The practical compliance test is simple to state and difficult to satisfy : if a regulator asked to see evidence of compliance for a given system within 24 hours, could you produce it? Organizations that cannot pass this test have a remediation gap; and the gap is typically not in the underlying work, but in documentation organization and retrieval readiness.
This distinction matters because it changes the remediation priority. Organizations that interpret a failed test as evidence that they have not done enough compliance work will spend the next quarter producing more documents. Organizations that interpret it as evidence of a retrieval infrastructure gap will spend the next quarter building document control, retention policies, version histories, and access controls. Only the second response actually closes the gap.
The test is a diagnostic, not speculative. National competent authorities under the EU AI Act have the power to request documentation from providers of high-risk AI systems as part of their market surveillance and post-market monitoring functions. The format of those requests varies by jurisdiction, but the substantive expectation is consistent : produce the technical documentation, the conformity assessment evidence, the post-market monitoring data, and the supporting records. Organizations that cannot produce these on demand are presumed ( under the standard regulatory logic the Act applies) to be non-compliant. The reverse presumption does not apply. Poor documentation creates a presumption of non-compliance even when the underlying controls exist.
Seven Evidence Types
The EU AI Act does not treat compliance as a single document. It treats compliance as an evidence architecture ; a structured set of artifacts, each supporting a specific obligation, each retrievable on demand. Seven evidence types recur across the obligation surface.
Each evidence type has a distinct purpose and a distinct retention requirement. Each must be retrievable independently. And each must be supported by a document control regime that makes retrieval possible ; version history, access permissions, integrity guarantees, and a known location within the organization’s information architecture. A controlled document that exists in a shared drive with no version history is not evidence ; it is a draft.
The six-month minimum on monitoring data and audit trails is a floor, not a ceiling. Organizations with related legal holds (litigation, regulatory investigations, in-flight complaints) must extend retention accordingly. The clock starts from the moment of the recorded event, not from system deployment. Log integrity is the deployer’s responsibility : logs must be tamper-evident and retained in a manner that preserves their evidentiary value.
Annex IV’s Primary Evidence
Of the seven evidence types, technical documentation under Annex IV is the primary evidence that conformity assessment bodies use to verify compliance. The Annex IV content areas are seven in number, and each must be present and current :
1. System description and intended purpose : what the system is, what it does, and what it is intended not to do.
2. System architecture : design decisions, algorithms, training approaches.
3. Training methodology and training data characteristics : provenance, representativeness, bias examination.
4. Testing procedures and results : accuracy, robustness, cybersecurity.
5. Known limitations and acceptable risks : what the system cannot do and what residual risks are formally accepted.
6. Cybersecurity measures : protections against unauthorized access, modification, and adversarial input.
7. Post-market monitoring plan : how the provider will measure performance and capture incidents after deployment.
The “kept up-to-date” requirement is the one many organizations miss. Many treat technical documentation as a one-time deliverable produced at launch. The Act requires documentation to reflect every significant system change ; which means the documentation process must be wired into the change management process, not adjacent to it. Every model update, every deployment configuration change, every scope expansion should trigger a documentation review and revision if the change affects any of the seven content areas. Organizations that cannot answer “when was this section of the Annex IV documentation last updated, and what change triggered that update?” have a documentation change management gap ; and that gap is auditable.
Self-Assessment Caveat
For most high-risk AI systems listed in Annex III, providers may use self-assessment : they declare conformity based on their own documentation and testing. There is no third-party auditor reviewing the work before the declaration is signed.
This makes the documentation burden more critical, not less. The self-assessment regime shifts the entire evidence production responsibility onto the provider. The regulator does not arrive before the system goes live ; the regulator arrives after ; and reviews the documentation that exists at that point. If the documentation is incomplete, outdated, or absent, the provider cannot retroactively demonstrate compliance. The work must be documented as it is performed, not reconstructed after the fact.
There are three places where the self-assessment trap closes most visibly :
Static versus iterative risk management. Article 9 requires the risk management process to be iterative : updated when the system changes, when new risks emerge, when residual risks shift. Organizations that conducted a risk assessment at design time and have not revisited it have a static risk record that does not satisfy Article 9 for any system whose behavior has changed since deployment. For agentic systems, where post-deployment behavioral change is the norm rather than the exception, a static risk record is structurally non-compliant.
Passive versus active post-market monitoring. Article 17 requires post-market monitoring that is active : that proactively measures system performance against defined KPIs. A monitoring regime that only collects user complaints is passive. The Act requires instrumentation : the system must emit performance data that can be collected, aggregated, and analyzed. Organizations whose monitoring consists of a ticketing queue do not satisfy Article 17.
Paper versus operational human oversight. Article 14 requires oversight to be effective. Oversight that exists on paper but is practically inaccessible does not satisfy the requirement. A human overseer who cannot realistically review system outputs before they take effect has not fulfilled Article 14 ; even if the oversight policy is documented and the role is assigned. The test is operational, not documentary.
What 24-Hour Retrieval Requires
Translating the test into operational terms: “retrievable within 24 hours” means a designated person can locate, verify the integrity of, and present each evidence artifact to a regulator within that window. The mechanism that makes this possible is document control : the set of procedures and systems that govern how documents are created, reviewed, approved, distributed, retained, and retrieved.
Version history ensures the current version of a controlled document is identifiable and that prior versions are preserved. Retention policies ensure that nothing the regulator may request has been prematurely deleted. Access permissions ensure that the people responsible for retrieval can actually retrieve. Integrity guarantees (cryptographic signatures, write-once storage, audit logs of access) ensure that what is retrieved has not been altered in a way that would compromise its evidentiary value.
The structural answer to the 24-hour test is the quality management system required by Article 16. Organizations without a QMS should implement one before attempting to comply with Articles 9 through 15 individually. The QMS is the system that ties the other requirements together. ISO/IEC 42001:2023 (the AI management system standard) is presumed to satisfy Article 16, which means an organization with ISO 42001 certification has the structural foundation that makes the 24-hour test passable.
Organizations without an existing management system face a different problem. They cannot satisfy Article 16 by writing a QMS document ; the regulation requires a system, not a document. A QMS without operational change management, without a functioning internal audit program, without management review that produces decisions, is documentation theatre. It does not satisfy Article 16 ; and it will not survive contact with a regulator who asks to see the change records, the audit findings, and the management review minutes.
Agentic AI: Evidence Categories That Don’t Yet Exist
The seven standard evidence types assume a static system whose behavior is fully characterized by its design documentation. For agentic systems (systems capable of autonomous reasoning, planning, and action) that assumption breaks.
Agentic technical documentation must specify the autonomy boundary : what the agent may and may not do without human intervention. It must document the human oversight mechanisms and escalation paths that apply when the agent encounters situations outside that boundary. It must capture the known limitations of multi-agent coordination and open-world operation ; situations the design did not and could not anticipate.
Agentic logging must capture qualitatively different content. Decision points and reasoning traces, tool calls and external system interactions, agent-to-agent messages and delegation events, human override events, goal state changes : none of these are part of conventional logging regimes, and most existing log architectures do not produce them by default. The six-month retention floor still applies, but the content of those logs is qualitatively different and not yet standardized across providers. Organizations building agentic systems must design their logging architecture for retrieval from the start : adding reasoning traces to an existing log pipeline as an afterthought produces logs that exist but cannot answer the questions a regulator will ask.
This is the operational reality that the next two articles in this series address. Article 9 examines the thirteen governance frameworks that organizations are using to organize their thinking ; including the few that address runtime governance for agentic systems. Article 10 examines what a real AI governance program looks like when it moves from assessment to action. The 24-hour test is the bridge between them : it converts the question from “what should we be doing?” into “what can we produce on demand?” ; and that conversion is what separates organizations that survive regulatory scrutiny from those that do not.
Download our free EU AI Act Compliance Evidence Checklist : System-Level
Next: The 13 AI Governance Frameworks — Comparing Maturity, Scope, and What Actually Differentiates Them