Skip to main content

Core Concepts

Projects​

A Project represents a monitored AI system. Noah supports two types:

  1. ENDPOINT Projects: Traditional ML models with test runs and metrics
  2. LLM Projects: Language model monitoring with proxy-based tracking

Each project can have:

  • Multiple test runs with metrics
  • Alert rules for automated monitoring
  • Golden datasets for drift detection
  • Evidence packs for compliance

Organizations​

An Organization is the top-level tenant entity:

Features:

  • Complete data isolation
  • Billing and subscription management
  • Team member management with roles
  • Integration configurations

Roles & Permissions​

Noah implements fine-grained RBAC with four roles:

RolePermissions
AdministratorFull access including team management and billing
EngineerCreate/edit projects, manage alerts, cannot invite users
Compliance OfficerView projects, manage evidence packs, read-only for technical features
ViewerRead-only access to all data

Permission Matrix:

Runs​

A Run represents a single test execution or monitoring session:

Contains:

  • Metrics (accuracy, precision, recall, F1, robustness, drift)
  • Execution metadata (start/end times, status)
  • Links to evidence packs

Authentication & Security​

Authentication Methods​

Noah supports two authentication approaches for different use cases:

1. Session-Based Authentication (Web UI)​

  • Email/password login with secure HTTP-only cookies
  • Automatic session management with 30-day expiration
  • Optional Two-Factor Authentication (2FA) via TOTP or SMS
  • Session monitoring with device tracking and location

2. API Key Authentication (Programmatic)​

  • Format: ctf_[32-character-string]
  • SHA-256 hashed storage for security
  • Organization-scoped with fine-grained permissions
  • Configurable expiration and usage limits
  • Real-time usage tracking and audit logs

Two-Factor Authentication (2FA)​

Enhance account security with TOTP (Time-based One-Time Password) or SMS verification:

TOTP Setup (Recommended):

  1. Download authenticator app (Google Authenticator, Authy, 1Password)
  2. Scan QR code or enter secret key manually
  3. Enter 6-digit verification code
  4. Save backup codes securely

SMS Setup:

  1. Enter and verify phone number
  2. Receive 6-digit code via SMS
  3. Enter code to complete setup
  4. Save backup codes

Backup Codes: Each account receives 5 single-use backup codes for recovery if 2FA device is unavailable.

Session Management​

Monitor and control active sessions across devices:

  • Current Session: View active device, browser, location, IP, and last activity
  • All Sessions: List all active sessions with device details
  • Terminate: End individual sessions or all other sessions
  • Activity Log: Track all login attempts with status, location, and timestamp

Security Alerts:

  • 3+ failed login attempts trigger email notification
  • New device login requires email confirmation
  • Login from new country requires 2FA verification

API Key Security​

Best Practices:

  1. Environment-Specific Keys: Use separate keys for dev, staging, production
  2. Secrets Management: Store keys in AWS Secrets Manager, HashiCorp Vault, or Kubernetes Secrets
  3. Rotation Schedule: Rotate keys every 90 days
  4. Monitor Usage: Track requests, last used timestamp, and usage patterns
  5. Revoke Compromised: Instantly revoke keys and terminate all sessions

Key Permissions:

  • Read projects
  • Write metrics
  • Manage alerts (Admin only)
  • Delete data (Admin only)

Team Management​

Team Roles & Workflows​

Inviting Team Members​

Process (Administrator only):

  1. Navigate to Settings → Team Management
  2. Click "Invite Member"
  3. Enter email address
  4. Select role (Administrator, Engineer, Compliance Officer, Viewer)
  5. Add optional personal message
  6. Send invitation

Invitation Flow:

  • Invitee receives email with accept/decline link
  • Link expires in 7 days
  • Upon acceptance, user joins organization with assigned role
  • Administrator receives confirmation notification

Managing Team Members​

Available Actions:

  • Change Role: Modify permissions (shows impact before confirming)
  • Remove Member: Revoke access immediately (terminates sessions, preserves audit logs)
  • Resend Invitation: Send new invitation email for pending invites
  • Cancel Invitation: Delete pending invitation

Switching Organizations​

Users can belong to multiple organizations with different roles:

Organization Switcher:

  • Dropdown menu shows all organizations
  • Displays current role in each organization
  • Switch between organizations instantly
  • Dashboard updates to show selected organization's data
  • Independent permissions per organization

Organization Settings​

Profile Configuration:

  • Organization name and URL slug
  • Logo upload (PNG/JPG, max 2MB)
  • Description and industry
  • Company size

Preferences:

  • Default notification channels
  • Data retention period
  • API rate limits by tier
  • Time zone and date/time formats

Danger Zone:

  • Transfer organization ownership
  • Delete organization (requires confirmation, irreversible)