Skip to main content

Evidence Packs & Compliance

Evidence Pack Structure​

Noah generates comprehensive evidence packs that serve as audit-ready documentation for regulatory compliance. Each evidence pack is a cryptographically signed, timestamped document that contains complete records of your AI system's behavior, performance, and compliance controls. These packs are designed to map directly to major regulatory frameworks, making it easy to demonstrate compliance during audits while saving countless hours of manual documentation effort.

The evidence pack structure is organized into seven main components, each serving a specific purpose in the compliance documentation process:

Metadata contains essential information about the project, including the project name, description, time period covered, selected compliance framework, and generation timestamp. This section provides auditors with immediate context about what the evidence pack covers and its scope.

Test Results include detailed records of all test runs during the covered period, with complete pass/fail statistics, error logs for failed tests, and performance trends over time. Each test run is documented with its configuration, inputs, outputs, and all collected metrics.

Metrics Data encompasses the comprehensive performance, quality, cost, and security metrics collected during monitoring. This includes success rates, latency measurements, token usage, costs, drift scores, readability metrics, sentiment analysis, and PII detection counts—providing complete visibility into system behavior.

Alert History documents all triggered alerts, their resolution status, response times, and root cause analyses. This demonstrates your organization's active monitoring and incident response capabilities, showing that issues are detected and resolved promptly.

Compliance Mapping ties everything together by mapping each piece of evidence to specific regulatory requirements. This section includes framework requirements, evidence cross-references, gap analysis identifying any missing documentation, and remediation plans for addressing gaps.

Cryptographic Proof ensures the authenticity and integrity of the evidence pack through SHA-256 hashing, digital signatures using industry-standard Cosign, and trusted timestamps. This makes the evidence pack legally defensible and verifiable by third parties.

Compliance Framework Mappings​

EU AI Act Article 15​

Requirements → Evidence Mapping:

Article 15.1 - General Description​

  • (a) System Purpose

    • Evidence: Project description, use case documentation
    • Location: Metadata section

  • (b) Development Process

    • Evidence: Version history, change logs
    • Location: Git commits, migration history

  • (c) Design Specifications

    • Evidence: Model configuration, parameters
    • Location: runner_config, model settings

Article 15.2 - Data Governance​

  • (a) Training Data

    • Evidence: Golden dataset, baseline data
    • Location: golden_dataset field

  • (b) Data Quality

    • Evidence: Data validation logs
    • Location: Upload validation results

  • (c) Bias Testing

    • Evidence: Robustness test results
    • Location: Test runs with robustness_score

Article 15.3 - Testing & Validation​

  • (a) Test Procedures

    • Evidence: Test configuration, methodology
    • Location: Test runs metadata

  • (b) Test Results

    • Evidence: All metrics from runs
    • Location: Metrics table, aggregated results

  • (c) Performance Metrics

    • Evidence: Success rate, latency, accuracy
    • Location: task_success_rate, task_e2e_latency

Article 15.4 - Risk Management​

  • (a) Identified Risks

    • Evidence: Alert configurations
    • Location: Alert rules table

  • (b) Mitigation Measures

    • Evidence: PII redaction, content filters
    • Location: PII detection logs, filter config

  • (c) Residual Risks

    • Evidence: Known limitations documentation
    • Location: Project notes, limitations field

Article 15.5 - Human Oversight​

  • (a) Oversight Measures

    • Evidence: Alert acknowledgments
    • Location: Alert history with user actions

  • (b) Intervention Capability

    • Evidence: Manual override logs
    • Location: Audit trail of manual actions

Coverage: 95% (19/20 requirements)
Gaps: 1 requirement needs manual documentation

DORA (Digital Operational Resilience Act)​

Article 6 - ICT Systems & Tools Identification​

  • Evidence: System inventory, dependencies
  • Location: Projects table, integration configs
  • Status: Compliant ✓

Article 7 - Protection & Prevention​

  • Evidence: Security measures, access controls
  • Location: RBAC policies, API key management
  • Status: Compliant ✓

Article 8 - Detection​

  • Evidence: Monitoring systems, alert rules
  • Location: Metrics collection, alert engine
  • Status: Compliant ✓

Article 9 - Response & Recovery​

  • Evidence: Incident logs, recovery procedures
  • Location: Alert history, runbook documentation
  • Status: Compliant ✓

Article 10 - Learning & Evolving​

  • Evidence: Post-incident reviews
  • Location: Alert resolution notes
  • Status: Compliant ✓

Article 11 - Testing​

  • Evidence: Test execution logs
  • Location: Runs table with timestamps
  • Status: Compliant ✓

Article 13 - Communication​

  • Evidence: Notification logs
  • Location: Integration delivery logs
  • Status: Compliant ✓

Coverage: 100% (7/7 core requirements)

Generating Evidence Packs​

Automatic Generation​

Configuration Options​

Auto-Generation Settings:

  • Generate evidence pack after each test run
  • Include detailed metrics in pack
  • Attach alert history
  • Sign with Cosign
  • Include timestamp from trusted source

Compliance Framework Selection:

  • EU AI Act Article 15
  • DORA (Digital Operational Resilience)
  • NIS2 (Network & Information Security)
  • OECD AI Principles
  • Custom Framework

Retention Policy:

  • Keep evidence packs for: 1 year
  • Archive after: 90 days

Export Options:

  • Default Format: PDF
  • Available: PDF, JSON, PowerPoint, Word

Evidence Pack Contents​

PDF Structure (50-100 pages):

1. COVER PAGE​

  • Project Name
  • Time Period
  • Generation Date
  • Compliance Framework
  • Digital Signature Hash

2. EXECUTIVE SUMMARY (2-3 pages)​

  • Project Overview
  • Key Findings
  • Compliance Status
  • Recommendations

3. TECHNICAL DETAILS (5-10 pages)​

  • Model Information
  • Configuration Settings
  • Infrastructure Details
  • Integration Points

4. TEST RESULTS (20-30 pages)​

  • Run Summary
  • Metrics Analysis
  • Pass/Fail Statistics
  • Performance Trends
  • Charts and Graphs

5. QUALITY METRICS (10-15 pages)​

  • Content Drift Analysis
  • Language Quality
  • Sentiment Analysis
  • Readability Scores

6. SECURITY & COMPLIANCE (10-15 pages)​

  • PII Detection Results
  • Content Safety Analysis
  • Access Control Audit
  • Incident Response

7. ALERT HISTORY (5-10 pages)​

  • Triggered Alerts
  • Resolution Status
  • Response Times
  • Root Cause Analysis

8. COMPLIANCE MAPPING (10-15 pages)​

  • Framework Requirements
  • Evidence Cross-Reference
  • Gap Analysis
  • Remediation Plan

9. APPENDICES (10-20 pages)​

  • Detailed Metrics Tables
  • Configuration Files
  • API Documentation
  • Glossary of Terms

10. VERIFICATION​

  • SHA-256 Hash
  • Digital Signature
  • Timestamp
  • Verification Instructions

Verifying Evidence Packs​

Command Line Verification​

# Download evidence pack
wget https://app.hollanoah.com/evidence/pack-abc123.pdf

# Verify hash
sha256sum pack-abc123.pdf
# Output: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

# Compare with signed hash from metadata
cat pack-abc123-metadata.json | jq .signed_hash
# Should match: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

# Verify Cosign signature
cosign verify --key noah-public.key pack-abc123.pdf
# Output: Verified OK

Web Verification Process​

Steps:

  1. Upload evidence pack or enter Pack ID
  2. System calculates hash
  3. Compares with stored hash
  4. Verifies digital signature
  5. Checks timestamp validity
  6. Confirms document integrity

Verification Results:

  • Hash Match: ✓ Confirmed
  • Signature: ✓ Valid
  • Timestamp: ✓ Within validity period
  • Integrity: ✓ Document unmodified

Use Cases​

Use Case 1: Financial Services Compliance​

Scenario: Bank deploying customer service chatbot must comply with regulations.

Implementation:

  1. Create LLM project for chatbot
  2. Upload golden dataset with approved responses
  3. Enable PII detection and redaction
  4. Set strict drift thresholds (0.5)
  5. Configure critical alerts for:
    • PII leakage in outputs
    • Drift exceeding threshold
    • Response latency > 3s
  6. Generate monthly evidence packs for audits
  7. Map to regulatory frameworks (DORA, NIS2)

Results: 100% PII redaction, audit-ready documentation, 99.8% compliance.

Use Case 2: Healthcare AI Assistant​

Scenario: Medical information chatbot requiring high accuracy and safety.

Implementation:

  1. Strict robustness testing for medical misinformation
  2. PII detection for patient data (HIPAA compliance)
  3. Content safety filters for harmful advice
  4. Evidence packs for regulatory audits
  5. Real-time alerts for refusal failures
  6. Human-in-loop for critical queries

Results: Zero data breaches, FDA audit compliance, 99.9% accuracy on medical queries.

Use Case 3: Government AI Systems​

Scenario: Public sector AI requiring transparency and accountability.

Implementation:

  1. Complete audit trails for all decisions
  2. Evidence packs mapped to EU AI Act
  3. Bias testing with diverse datasets
  4. Public-facing compliance reports
  5. Regular third-party audits
  6. Citizen feedback integration

Results: Full transparency, successful audits, public trust maintained.

Best Practices​

For Compliance Officers​

  1. Generate evidence packs monthly minimum
  2. Map to applicable frameworks early
  3. Review PII detection logs weekly
  4. Document all configuration changes
  5. Maintain audit trail of alerts
  6. Schedule regular compliance reviews
  7. Keep evidence packs for required retention period
  8. Prepare for audits with comprehensive documentation
  1. Understand evidence pack contents
  2. Verify cryptographic signatures
  3. Maintain chain of custody
  4. Document evidence handling procedures
  5. Prepare legal holds when necessary
  6. Coordinate with compliance officers
  7. Review framework mappings regularly

Documentation Standards​

  1. Use consistent naming conventions
  2. Version control all configurations
  3. Document decision rationale
  4. Maintain change logs
  5. Archive historical evidence
  6. Regular backup procedures
  7. Disaster recovery planning