Operations Deployment Guide

1. Overview

Purpose: Procedural guide for deploying and operating PCR.AI in production environments.

Audience: DevOps Engineers, System Administrators, Technical Leads

Related Documentation:

• SDS (Software Design Specification) - Design decisions and architecture
• STD (Software Test Documentation) - Test procedures and validation

Infrastructure Stack:

• Compute: Laravel Vapor (serverless on AWS Lambda)
• Database: AWS RDS (MySQL), DynamoDB (sessions)
• Storage: AWS S3 (file imports/exports)
• Authentication: AWS Cognito (user pools, SAML federation)
• Real-time: Pusher (WebSocket notifications)
• DNS/CDN: Cloudflare

---

2. Vapor Configuration

Vapor Secrets Management

For each project, create a Vapor secret for AWS credentials used by the Deployment Console.

Property	Format
Key Format	AWS_CREDENTIALS_{PROJECT_NAME_IN_SNAKE_CASE_AND_UPPER_CASE}
Value Format	{AWS_ACC_ID},{AWS_ACCESS_KEY},{AWS_SECRET_KEY}

Example:

• Project name: qa-viracor
• Secret key: AWS_CREDENTIALS_QA_VIRACOR
• Secret value: 123456789012,AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Important: Redeployment Required

After changing a Vapor environment secret:

1. A standard Vapor redeploy does NOT pick up secret changes
2. You must re-run the GitHub Action for the Deployment Console
3. Verify the new secret is active by checking application logs

# Trigger deployment via CLI
vapor deploy {environment}

# Or trigger via GitHub Actions workflow dispatch

---

3. S3 Bucket Provisioning

Naming Convention

{client-name}-pcrai-{environment}

Examples:

• viracor-pcrai-production
• labcorp-pcrai-staging
• quest-pcrai-qa

Standard Folder Structure

Each client bucket requires the following folder structure:

{client-bucket}/
├── toPcrai/           # Monitored import folder (S3 event trigger)
├── Processing/        # Temporary folder during import
├── Problem_Files/     # Failed imports with error logs
├── LIMS_Reports/      # LIMS export destination
├── archive/           # Successfully processed files
└── calibrations/      # DXAI calibration files

AWS CLI Commands for Folder Creation

# Set bucket name
BUCKET="viracor-pcrai-production"

# Create all standard folders
aws s3api put-object --bucket $BUCKET --key toPcrai/
aws s3api put-object --bucket $BUCKET --key Processing/
aws s3api put-object --bucket $BUCKET --key Problem_Files/
aws s3api put-object --bucket $BUCKET --key LIMS_Reports/
aws s3api put-object --bucket $BUCKET --key archive/
aws s3api put-object --bucket $BUCKET --key calibrations/

# Verify folder structure
aws s3 ls s3://$BUCKET/

Event Notification Configuration

Configure S3 event notifications for the toPcrai/ folder to trigger file processing:

# Create notification configuration JSON
cat > s3-notification.json << 'EOF'
{
  "QueueConfigurations": [
    {
      "QueueArn": "arn:aws:sqs:us-east-1:{account-id}:{queue-name}",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "toPcrai/"
            }
          ]
        }
      }
    }
  ]
}
EOF

# Apply configuration
aws s3api put-bucket-notification-configuration \
  --bucket $BUCKET \
  --notification-configuration file://s3-notification.json

IAM Policy Requirements

Attach the following policy to the Vapor Lambda execution role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::{bucket-name}",
        "arn:aws:s3:::{bucket-name}/*"
      ]
    }
  ]
}

---

4. Multi-Site S3 Setup

For clients with multiple laboratory sites, create site-level subfolders within the standard structure.

Site Folder Structure

{client-bucket}/
├── toPcrai/
│   ├── site-austin/
│   ├── site-chicago/
│   └── site-boston/
├── Processing/
│   ├── site-austin/
│   ├── site-chicago/
│   └── site-boston/
├── Problem_Files/
│   ├── site-austin/
│   ├── site-chicago/
│   └── site-boston/
├── LIMS_Reports/
│   ├── site-austin/
│   ├── site-chicago/
│   └── site-boston/
└── archive/
    ├── site-austin/
    ├── site-chicago/
    └── site-boston/

Site Provisioning Commands

BUCKET="viracor-pcrai-production"
SITE="site-austin"

# Create site subfolders in each standard folder
aws s3api put-object --bucket $BUCKET --key toPcrai/$SITE/
aws s3api put-object --bucket $BUCKET --key Processing/$SITE/
aws s3api put-object --bucket $BUCKET --key Problem_Files/$SITE/
aws s3api put-object --bucket $BUCKET --key LIMS_Reports/$SITE/
aws s3api put-object --bucket $BUCKET --key archive/$SITE/

# Verify site folders
aws s3 ls s3://$BUCKET/toPcrai/

---

5. DynamoDB Session Table

Auto-Provisioning via Laravel Vapor

Laravel Vapor automatically provisions and configures DynamoDB for session storage when SESSION_DRIVER=dynamodb is set.

Table Configuration (Vapor-Managed)

Property	Value
Table Name	{project}-{environment}-sessions
Primary Key	id (String)
TTL Attribute	expires (enabled)
Billing Mode	On-demand (PAY_PER_REQUEST)

Examples:

• pcrai-production-sessions
• pcrai-staging-sessions

Session Behavior

• Sessions stored as JSON documents in DynamoDB
• TTL automatically cleans expired sessions
• Single-device enforcement via Cognito GlobalSignOut (not DynamoDB-level)

Verification Command

# Verify session table exists and is configured correctly
aws dynamodb describe-table --table-name pcrai-production-sessions

# Check table status
aws dynamodb describe-table \
  --table-name pcrai-production-sessions \
  --query 'Table.{Name:TableName,Status:TableStatus,TTL:TimeToLiveDescription}'

No Manual Setup Required: Vapor handles table creation, IAM roles, and environment variable injection during deployment.

---

6. Cognito User Pool Setup

Step 1: Create New User Pool

1. Navigate to AWS Cognito Console
2. Click "Create user pool"
3. Use default settings for initial configuration

Step 2: Configure Authentication

1. Select sign-in options (email, username)
2. Configure password policy (minimum 8 characters, mixed case, numbers, symbols)
3. Enable MFA if required by client security policy

Step 3: Add Custom Attributes

1. Navigate to "Sign-up experience" > "Custom attributes"
2. Click "Add custom attribute"
3. Add the following custom attribute:

Attribute Name	Type	Mutable
UserGroup	String	Yes

Step 4: Configure App Client

1. Navigate to "App integration" > "App clients"
2. Create new app client
3. Configure OAuth 2.0 settings:
   • Allowed callback URLs: https://{domain}/login/samlcallback
   • Allowed sign-out URLs: https://{domain}/logout
   • OAuth 2.0 grant types: Authorization code grant
   • OpenID Connect scopes: openid, email, profile

Step 5: Create SAML Identity Provider (if federated SSO required)

1. Navigate to "Sign-in experience" > "Federated identity provider sign-in"
2. Click "Add identity provider" > "SAML"
3. Upload client's SAML metadata XML
4. Configure attribute mapping:

SAML Attribute	User Pool Attribute
email	email
given_name	given_name
family_name	family_name
custom:group	custom:UserGroup

Step 6: Review and Create

1. Review all settings
2. Create the user pool
3. Note the User Pool ID and App Client ID for environment configuration

---

7. Domain URL Changes

When changing the domain URL for an environment, update the following components in order:

Step 1: Vapor Deployment Manifest

Update vapor.yml:

environments:
  production:
    domain: new-domain.pcrai.com

Step 2: Cloudflare DNS

1. Navigate to Cloudflare DNS settings
2. Add or update CNAME record:

Type	Name	Target	Proxy Status
CNAME	new-domain	{vapor-cloudfront-distribution}.cloudfront.net	Proxied

Step 3: Cognito Callback URLs

1. Navigate to Cognito User Pool > App clients
2. Update callback URLs:
   • https://new-domain.pcrai.com/login/samlcallback
3. Update sign-out URLs:
   • https://new-domain.pcrai.com/logout

Step 4: Laravel Environment Variables

Update in Vapor/Deployment Console:

APP_URL=https://new-domain.pcrai.com
AWS_COGNITO_CALLBACK_URL=https://new-domain.pcrai.com/login/samlcallback
AWS_SUPER_ADMIN_COGNITO_CALLBACK_URL=https://new-domain.pcrai.com/login/samlcallback

Step 5: Redeploy Application

vapor deploy production

Step 6: Verify

# Check application responds
curl -I https://new-domain.pcrai.com/api/health

# Verify SSL certificate
openssl s_client -connect new-domain.pcrai.com:443 -servername new-domain.pcrai.com </dev/null 2>/dev/null | openssl x509 -noout -dates

Cognito Domain Deletion and Recreation

When changing an environment's domain, the existing Cognito domain must be deleted and recreated (not just edited). In the Cognito Console under "App integration" > "Domain":

1. Actions > Delete Cognito Domain to remove the old domain
2. Actions > Create Custom Domain to create the new one

This is a separate step from updating the callback/sign-out URLs in the App Client settings.

Video Walkthroughs

Internal walkthrough recordings for the domain change procedure (SharePoint):

• Change domain and deploy -- Deployment Console manifest update and redeploy
• Cloudflare DNS update -- CNAME record update with new domain and CloudFront target
• Update Cognito references -- Domain deletion/recreation and callback URL changes
• Update APP_URL in Vapor env -- Environment variable update and redeploy

---

8. Deployment Checklist

Pre-Deployment

#	Check	Action	Verify
1	Code Review	All PRs merged and approved	GitHub PR status shows approved
2	Tests Passing	CI/CD pipeline green	GitHub Actions all green
3	Environment Variables	All required vars set	Vapor console shows no missing vars
4	Secrets Updated	API keys, credentials current	Vapor secrets reviewed
5	Database Migrations	Migrations reviewed for breaking changes	Migration files checked
6	Backup	Database snapshot taken	AWS RDS console shows recent snapshot

Deployment Steps

Step 1: Initiate Deployment

# Via CLI
vapor deploy production

# Or via Deployment Console UI
# Navigate to: Deployment Console > Project > Deploy

Step 2: Monitor Deployment

• Watch Vapor console for deployment progress
• Check Lambda function deployment status
• Monitor for any timeout or memory errors

Step 3: Post-Deployment Validation

# Verify application health endpoint
curl https://pcrai.example.com/api/health

# Expected response:
# {"status":"ok","version":"v1.2.3","environment":"production"}

# Check application logs for errors
vapor logs production --filter="ERROR"

Step 4: Smoke Test

Perform the following manual checks:

• Login as test user (both native and SSO if applicable)
• Navigate to run file list - verify data loads
• Upload a test file via import
• Verify real-time notifications (Pusher) working
• Check audit log for login event

---

9. Rollback Procedures

Via Vapor Console

1. Navigate to Vapor > Project > Environment > Deployments
2. Locate the previous stable deployment (check deployment timestamp)
3. Click the Rollback button
4. Confirm the rollback action
5. Monitor rollback progress

Via CLI

# Rollback to previous deployment
vapor rollback production

# Rollback to specific deployment ID
vapor rollback production --deployment={deployment-id}

Database Migration Rollback

Important: Database migrations are NOT automatically rolled back with code rollback.

If migration rollback is required:

# Connect to production environment
vapor shell production

# Rollback last migration batch
php artisan migrate:rollback

# Rollback specific number of migrations
php artisan migrate:rollback --step=2

# Check migration status
php artisan migrate:status

Rollback Verification

After rollback:

# Verify application version
curl https://pcrai.example.com/api/health | jq '.version'

# Check application functionality
vapor logs production --recent

Emergency Contacts

• For production incidents, follow incident response runbook
• Escalate to Technical Lead if rollback fails
• Contact AWS Support for infrastructure-level issues

---

Appendix A: Environment Variable Reference

Important Notes

• Do not change variables via Vapor interface - Use Deployment Console Environment Variable Section instead
• Vapor deployments are immutable - after updating variables, you must redeploy for changes to take effect
• Individual variables may not exceed 2000 characters
• Secrets may not exceed 4096 characters (AWS Parameter Store limitation)

Application Core

Variable	Default	Security	Notes
APP_NAME	Pcr.ai	Standard	Application display name
APP_ORGANIZATION	Pcr.ai	Standard	Organization name
APP_ENV	production	Standard	Environment identifier
APP_KEY	(generate)	Restricted	Generate using php artisan key:generate
APP_DEBUG	false	Standard	Must be false in production
APP_URL	(required)	Standard	Full domain URL with https://
APP_VERSION	(auto)	Standard	Filled automatically during deployment

Database

Variable	Default	Security	Notes
DB_AUDIT_CONNECTION	mysql_audit	Standard	Audit database connection name
DB_AUDIT_HOST	(required)	Restricted	From Vapor database configuration
DB_AUDIT_PASSWORD	(required)	Restricted	From Vapor database configuration

Session

Variable	Default	Security	Notes
SESSION_DRIVER	dynamodb	Standard	Use DynamoDB for serverless
SESSION_LIFETIME	12960	Standard	Session lifetime in minutes (9 days)

Pusher (Real-time)

Variable	Default	Security	Notes
PUSHER_APP_ID	(required)	Restricted	Pusher application ID
PUSHER_APP_KEY	(required)	Restricted	Pusher application key
PUSHER_APP_SECRET	(required)	Restricted	Pusher application secret
PUSHER_APP_CLUSTER	mt1	Standard	Pusher cluster region
VUE_APP_PUSHER_APP_KEY	(required)	Restricted	Frontend Pusher key
VUE_APP_PUSHER_APP_SECRET	(required)	Restricted	Frontend Pusher secret
VUE_APP_PUSHER_APP_CLUSTER	mt1	Standard	Frontend cluster region

Cognito - Super Admin User Pool

Variable	Default	Security	Notes
AWS_SUPER_ADMIN_COGNITO_CLIENT_ID	(required)	Restricted	Super admin app client ID
AWS_SUPER_ADMIN_COGNITO_CLIENT_SECRET	(required)	Restricted	Super admin app client secret
AWS_SUPER_ADMIN_USER_POOL_ID	(required)	Restricted	Super admin user pool ID
AWS_SUPER_ADMIN_COGNITO_USER_POOL_DOMAIN_NAME	(required)	Restricted	Cognito domain name
AWS_SUPER_ADMIN_COGNITO_CALLBACK_URL	${APP_URL}/login/samlcallback	Standard	SAML callback URL
AWS_SUPER_ADMIN_COGNITO_REGION	us-east-1	Standard	AWS region

Cognito - Regular User Pool

Variable	Default	Security	Notes
AWS_COGNITO_CLIENT_ID	(required)	Restricted	Regular app client ID
AWS_COGNITO_CLIENT_SECRET	(required)	Restricted	Regular app client secret
AWS_USER_POOL_ID	(required)	Restricted	Regular user pool ID
AWS_COGNITO_USER_POOL_DOMAIN_NAME	(required)	Restricted	Cognito domain name
AWS_COGNITO_CALLBACK_URL	(required)	Standard	OAuth callback URL
AWS_COGNITO_REGION	(required)	Standard	AWS region
CORS_ALLOWED_ORIGINS	(required)	Standard	Comma-separated allowed origins

Monitoring & Rate Limiting

Variable	Default	Security	Notes
VUE_APP_SENTRY_DSN	(optional)	Standard	Sentry error tracking DSN
VUE_APP_SENTRY_REPLAY_SESSION_SAMPLE_RATE	0.1	Standard	Session replay sample rate
VUE_APP_SENTRY_REPLAY_ON_ERROR_SAMPLE_RATE	1.0	Standard	Error replay sample rate
VUE_APP_IDLE_TIMEOUT	1440	Standard	Client idle timeout (minutes)
RATE_LIMIT_MAX_ATTEMPTS	20	Standard	API rate limit per minute

Security

Variable	Default	Security	Notes
IP_WHITELIST	127.0.0.*,	Restricted	Comma-separated IP patterns
BLOCK_UNAUTHORIZED_IP	true	Restricted	Enable IP blocking
PREVENT_NATIVE_LOGIN	false	Restricted	Force SSO-only authentication

Email

Variable	Default	Security	Notes
MAIL_MAILER	sendgrid	Standard	Email driver
MAIL_FROM_NAME	Pcr.ai	Standard	Email sender name

Vapor Secrets (Highly Restricted)

These are stored in Vapor Secrets, not environment variables. After updating, redeploy is required.

Secret	Notes
PARSER_API_KEY	Parser service API key
PARSER_URL	Parser service URL
PARSER_VERSION	Parser API version (default: 1)
AWS_CLIENT_COGNITO_CLIENT_ID	Client Cognito app ID
AWS_CLIENT_COGNITO_CLIENT_SECRET	Client Cognito app secret
AWS_SUPER_ADMIN_ACCESS_KEY_ID	Super admin AWS access key
AWS_SUPER_ADMIN_SECRET_ACCESS_KEY	Super admin AWS secret key
AWS_SUPER_ADMIN_DEFAULT_REGION	AWS region (default: us-east-1)
DXAI_CALIBRATOR_API_URL	DXAI calibrator service URL
DXAI_CALIBRATOR_API_KEY	DXAI calibrator API key
DXAI_CALIBRATOR_API_VERSION	DXAI API version (default: 1)

Variable Categories Summary

Category	Count	Security Level
Application Core	7	Standard
Database	3	Restricted
Session	2	Standard
Pusher	7	Restricted
Cognito Super Admin	6	Restricted
Cognito Regular	7	Restricted
Monitoring	5	Standard
Security	3	Restricted
Email	2	Standard
Secrets	11	Highly Restricted
Total	53	-

---

Last Updated: 2026-01-25