Delivering Operational Excellence for Mission-Critical Aviation Software on AWS

About the client

The company, a subsidiary of TKH Airport Solutions, is a Canadian technology company specializing in airfield guidance and deicing management software for airports and airlines worldwide. Under their AIM Systems brand, the company develops and operates two mission-critical platforms: Icelink, a web-based information management system that orchestrates the full aircraft deicing workflow, integrating with airport operational databases (AODB, A-CDM, FIDS), GPS-tracking deicing vehicles, managing glycol usage and compliance reporting, and providing pilots with Hold Over Time data via a dedicated mobile application, and SmartPad, a bay management system that guides aircraft through centralized deicing facilities using Electronic Message Boards and optical guidance systems.

As of 2025, 3 of the 4 largest US airlines rely on Icelink to manage and optimize their deicing operations. The platform is deployed at airports including Vancouver International (YVR), Geneva Airport (GVA), Amsterdam Schiphol (AMS), and London Stansted (STN), among others. The company's entire AWS infrastructure is hosted in the US East (N. Virginia) region with a Pilot Light disaster recovery environment in US East (Ohio).

Challenges

Aircraft deicing is a safety-critical operation in commercial aviation: an aircraft that departs with ice contamination on its lifting surfaces presents a direct flight safety hazard. The software that coordinates this process, managing pad capacity, tracking vehicle movements, logging fluid usage, and exchanging data with airline systems in real time, must be available and performant during precisely the conditions that place the greatest demand on it: winter weather operations, often coinciding with high traffic volumes and tight turnaround windows. For the company, serving 3 of the 4 largest US airlines and major international airports across multiple continents, any degradation in platform availability or performance during active deicing operations carries direct consequences for flight schedules, ground crew coordination, and airline operational safety commitments.

By the second half of 2025, a Teracloud-led assessment identified a critical gap: the DevOps operating model supporting the Icelink and SmartPad platforms had not kept pace with the operational criticality of the workload. The infrastructure was technically sound at the component level, but lacked the observability depth, incident response structure, and operational governance required for a platform handling safety-sensitive aviation workflows at scale.

The assessment identified the following specific gaps:

Application-level observability was absent. While Amazon CloudWatch provided infrastructure metrics, there was no application performance monitoring or distributed tracing in place. The engineering team had no visibility into request-level performance, service dependencies, or latency behavior, meaning degradation in the Icelink platform could occur without detection until users reported it.
Alert thresholds were miscalibrated, producing noise rather than signal. Existing alerting configurations generated excessive alerts due to poorly tuned thresholds and a lack of KPI-based prioritization. The result was alert fatigue: engineers became desensitized to notifications, undermining the reliability of the incident detection process at precisely the moments when fast response was most critical.
Incident detection and resolution times were unacceptable for the platform's risk profile. With a Mean Time to Detection (MTTD) averaging 20 minutes and a Mean Time to Resolution (MTTR) averaging 10 hours and 21 minutes, the operational posture was reactive rather than proactive. For a platform coordinating aircraft deicing at major international airports, a 20-minute detection window and 10-hour resolution cycle represent significant exposure to service degradation during active operations.
Scaling behavior under peak load was inconsistent. During high-demand periods, precisely the winter weather events when the platform is most critical, infrastructure scaling required optimization. Reactive scaling configurations introduced temporary latency increases and delayed resource provisioning under sudden traffic spikes.
Access and identity governance lacked the structure required for a multi-account environment. As the environment had grown to span multiple AWS accounts, managing access across development, staging, and production had become increasingly complex, introducing risk of over-permissioned roles and limiting the ability to audit access systematically.

Prior to the engagement, these gaps were measurable: p95 API latency stood at 25ms, MTTD averaged 20 minutes, and MTTR averaged 10 hours and 21 minutes, a baseline that reflected the operational constraints described above and established the benchmark against which post-engagement improvements were measured.

Risk of not addressing this challenge

Operational safety: A platform coordinating deicing operations for 3 of the 4 largest US airlines and major international airports cannot sustain a 10-hour resolution window during winter operations. A prolonged outage or undetected degradation during active deicing could disrupt the coordination of safety-critical ground operations across multiple airline clients simultaneously.

SLA and commercial: Airline and airport clients operate under strict schedule commitments. Platform unavailability during peak winter operations translates directly into delays, SLA breaches, and reputational risk for JCAII across a client base that includes some of the world's busiest airports.

Operational scalability: Without structured observability, incident response, and access governance, the platform could not scale its client base without proportionally increasing operational risk, limiting JCAII's growth trajectory precisely as demand for digital deicing management was increasing globally.

Goals and Objectives

The company engaged Teracloud to close the operational maturity gap identified in the assessment and align the DevOps practices supporting the Icelink and SmartPad platforms with the safety-critical nature of the workload they serve. The engagement objectives were defined working backwards from a clear outcome: a production environment that could detect, surface, and resolve operational issues before they affected airline and airport operations, and that could scale its client base without proportionally increasing operational risk.

Business Objectives

Reduce incident detection and resolution times to levels appropriate for safety-critical aviation software. With a MTTD of 20 minutes and MTTR of 10 hours and 21 minutes at baseline, the objective was to implement monitoring and incident response infrastructure capable of detecting issues within minutes and resolving them within a timeframe consistent with the operational commitments made to airline and airport clients.

Establish end-to-end application observability. Infrastructure-level metrics alone were insufficient for a distributed platform handling real-time deicing coordination. The objective was to implement full-stack observability, combining infrastructure monitoring, application performance monitoring, distributed tracing, and structured log analysis, to give the engineering team complete visibility into platform behavior under all operational conditions.

Eliminate alert fatigue and establish a signal-based incident response model. The objective was to replace noisy, threshold-based alerting with KPI-aligned, prioritized alerts routed through a structured incident management workflow, ensuring that when an alert fires, it represents a meaningful operational event requiring action.

Build the operational foundation to scale the client base safely. As the company expanded its airline and airport client roster, each new deployment needed to operate within a governance model that enforced consistent security, access, and operational standards, without requiring proportional growth in engineering overhead.

Technical Objectives

Implement Datadog APM for distributed tracing and application performance monitoring. Instrument the Icelink platform services to capture request-level traces, latency distributions, service dependency maps, and error propagation paths, providing the application-level visibility that CloudWatch infrastructure metrics alone could not deliver.

Define and instrument workload health KPIs. Establish a structured set of KPIs, API latency (p95), application error rate, infrastructure utilization (CPU, memory), and service availability, with defined thresholds and data sources across both CloudWatch and Datadog, creating a measurable, objective baseline for platform health.

Calibrate alerting thresholds and implement PagerDuty for structured incident response. Replace miscalibrated alert configurations with KPI-aligned thresholds differentiated by severity. Integrate CloudWatch and Datadog alerts with PagerDuty to establish structured on-call schedules, escalation policies, and incident lifecycle management, ensuring every alert routes to the right person at the right time.

Implement CloudWatch Synthetics for continuous endpoint availability monitoring. Deploy synthetic canaries to validate Icelink platform availability from the outside in, providing an independent availability signal that does not depend on application instrumentation and can detect endpoint failures before users report them.

Optimize Kubernetes scaling for peak-demand events. Tune Horizontal Pod Autoscaler (HPA) and Karpenter node provisioning configurations to ensure the platform responds to traffic spikes, particularly during winter weather operations, without latency degradation or delayed resource provisioning.

Centralize identity governance via AWS IAM Identity Center. Implement role-based access control across all accounts through IAM Identity Center, with permission sets aligned to team roles and least-privilege policies enforced throughout, eliminating over-permissioned roles and establishing full auditability of access across the environment.

Establish a formal operational runbook aligned with platform KPIs. Document structured troubleshooting playbooks for each KPI-defined failure scenario, integrating Datadog APM, CloudWatch, and PagerDuty workflows, giving the on-call team a consistent, repeatable response framework for every operational event.

Solution and Execution

Teracloud's engagement with the company was structured as a DevOps maturity uplift across an existing production architecture, addressing the observability, incident response, scaling, and governance gaps identified in the assessment while maintaining continuity of service for the company's airline and airport clients. Work was delivered across five interconnected domains: observability and incident response, deployment automation, infrastructure as code governance, security and identity, and disaster recovery.

Observability and Incident Response: Teracloud deployed a full-stack observability model to replace the incomplete monitoring posture that had contributed to a 20-minute MTTD and 10-hour MTTR. Datadog APM was instrumented across Icelink platform services, providing distributed tracing, request-level performance monitoring, service dependency mapping, and centralized dashboards for real-time operational visibility. Application logs are structured, exported, and correlated with APM traces for root cause analysis. CloudWatch synthetic canaries were deployed to continuously validate endpoint availability from the outside in, independently of application instrumentation. Alert thresholds across both CloudWatch and Datadog were recalibrated against KPI-defined baselines, differentiating warning from critical severity and eliminating the noise that had previously caused alert fatigue. All alerts route through PagerDuty, which manages on-call scheduling, escalation policies, and incident lifecycle tracking from detection to resolution. A formal operational runbook (OPE-002) documents structured troubleshooting playbooks for each KPI-defined failure scenario, covering daily, weekly, and monthly operational tasks, giving the on-call team a consistent, repeatable response framework.

CI/CD Pipeline and Deployment Automation: Application changes flow through a fully automated pipeline: GitHub pull requests trigger GitHub Actions workflows that initiate AWS CodeBuild processes, executing unit tests, integration tests, Helm chart validation (lint and pre-commit hooks), and security vulnerability scanning via Dependabot and Amazon Inspector. Validated container images are built and published to Amazon ECR. Production deployments are managed via ArgoCD using Blue-Green deployment strategies through Argo Rollouts, enabling zero-downtime releases with automated rollback capability. A Slack bot integration allows the engineering team to trigger and monitor the production Blue-Green promotion workflow, with all deployment actions logged and auditable. Deployment progression follows a formal dev → staging → production pipeline, with a change management plan reviewed and approved before any production release. Post-deployment validation includes smoke tests and monitoring verification via Datadog dashboards and PagerDuty alert status.

Infrastructure as Code: All AWS infrastructure is defined, versioned, and managed using Terraform, maintained in GitHub repositories and subject to mandatory peer review through pull requests, the same review and approval workflow applied to application code. Terraform modules enforce reusability and consistency across resources, with environment-specific values managed via locals to prevent code duplication. All managed AWS resources are tagged to maintain IaC traceability. The DR infrastructure in us-east-2 is maintained in the same Terraform codebase as production, ensuring full environment parity and preventing configuration drift between regions.

Compute, Data, and Application Architecture: The backend is containerized and orchestrated by Amazon EKS, with EC2 worker nodes distributed across multiple Availability Zones. Karpenter dynamically provisions replacement nodes based on workload demand, and the Horizontal Pod Autoscaler (HPA) scales application pods in response to CPU, memory, or custom metrics, ensuring the platform handles peak winter operations traffic without latency degradation. Container images are stored in Amazon ECR private repositories, with one repository per microservice.
Aurora PostgreSQL serves as the primary relational database, deployed in multi-AZ configuration with read replicas and an Aurora Global Database cluster providing continuous cross-region replication to the DR environment. Amazon ElastiCache for Valkey provides an in-memory cache layer across multiple Availability Zones, synchronized with Aurora as the data source for high-performance query execution. AWS Secrets Manager handles database credentials with automatic rotation enabled. AWS Systems Manager Parameter Store manages environment variables and configuration.
Static frontend assets are served via Amazon CloudFront with an S3 bucket as origin, configured with an Origin Group failover to a secondary S3 bucket in the DR region, ensuring static content availability in the event of a regional outage. ALB and CloudFront endpoints are protected by AWS WAF and secured with TLS certificates via AWS Certificate Manager. Cloudflare provides DNS resolution and an additional CDN proxy layer for custom domain management.

Security and Identity: Identity management is centralized via AWS IAM Identity Center, with role-based permission sets assigned through groups, Developers, Power Developers, DevOps, Billing, QA, enforcing least-privilege access across all accounts. No shared credentials or direct IAM users are used; all access is federated through IAM Identity Center with temporary credentials. Legacy applications use IAM access keys with 90-day rotation; modern applications leverage IAM Roles Anywhere for secure credential management.
Encryption at rest is enforced using customer-managed KMS keys (CMKs) with automatic annual rotation across Aurora PostgreSQL, EBS, S3, ECR, ElastiCache Valkey, CloudWatch Logs, Secrets Manager, and Parameter Store. All internet-facing traffic is encrypted via HTTPS/TLS through CloudFront and ALB. AWS GuardDuty provides continuous threat detection including EKS runtime monitoring. Amazon Inspector assesses EC2 instances and ECR container images for vulnerabilities. AWS Security Hub centralizes findings across all accounts in the security account. AWS Security Lake aggregates and normalizes log telemetry in OCSF format across the organization for cross-account security analysis. CloudTrail is configured as an organization-wide trail with logs centralized in the security account, protected by SCPs that deny trail deletion or disabling across all accounts.

Multi-Account Governance and Disaster Recovery: The environment is structured under AWS Organizations with accounts separated by function, production, development/staging, and security, governed by Service Control Policies applied at the organization root. IAM Identity Center provides centralized federation across all accounts.
All infrastructure is replicated in us-east-2 following a Pilot Light disaster recovery strategy. Aurora Global Database provides near-real-time cross-region data replication, supporting a near-zero RPO. CloudFront Origin Groups and S3 cross-region replication maintain static content availability. In the event of a regional failure, the DR recovery workflow promotes the Aurora Global Database secondary cluster, deploys and scales EKS workloads, and updates traffic routing, targeting an RTO of 4 hours.

Architecture Overview

The following diagram illustrates the company's production AWS architecture deployed across multiple Availability Zones in US East (N. Virginia), with a Pilot Light disaster recovery environment in US East (Ohio).

Compute layer: Amazon EKS with EC2 worker nodes distributed across multiple Availability Zones; Karpenter for dynamic node provisioning; Horizontal Pod Autoscaler (HPA) for pod-level scaling; container images stored in Amazon ECR private repositories.

Data layer: Aurora PostgreSQL Global Database (multi-AZ, read replicas, cross-region replication to DR); Amazon ElastiCache for Valkey (multi-AZ in-memory cache); AWS Backup for automated backup management.

Content delivery: Amazon CloudFront with S3 as origin and Origin Group failover to DR region for static assets; ALB for backend traffic distribution.

Networking: Multi-tier VPC with public subnets (ALB), private compute subnets (EKS), and private database subnets (Aurora, ElastiCache); one NAT Gateway per AZ; Security Groups and Network ACLs enforcing least-privilege traffic segmentation.

Security: AWS WAF on CloudFront and ALB; IAM Identity Center SSO with MFA; KMS CMK encryption across all data stores; Secrets Manager; GuardDuty (including EKS runtime monitoring); Security Hub; Amazon Inspector; Security Lake (centralized log aggregation in OCSF format); CloudTrail organization trail.

CI/CD and IaC: GitHub Actions + CodeBuild → ECR → ArgoCD (Blue-Green via Argo Rollouts); Terraform managing all infrastructure via GitHub PR workflow.

Monitoring: CloudWatch (infrastructure metrics + synthetic canaries) + Datadog APM (distributed tracing + dashboards) → PagerDuty (incident management and escalation).

Disaster recovery: Pilot Light architecture in us-east-2; Aurora Global Database cross-region replication; CloudFront Origin Group failover; RTO 4 hours, RPO near-zero.

KPIs And Outcomes

The following key performance indicators were measured against baseline values captured prior to the engagement, reflecting the operational improvements achieved through Teracloud's DevOps implementation across the Icelink platform.

Mean Time to Detection, 85% Reduction (20 min → 3 min): The deployment of Datadog APM with KPI-calibrated alerting and PagerDuty integration reduced the average time between issue occurrence and incident creation from 20 minutes to 3 minutes. For a platform managing real-time deicing coordination at major international airports, this represents a fundamental shift in operational posture: issues are now surfaced to the engineering team before they propagate into user-facing failures or affect airline ground operations.

Mean Time to Resolution, 91% Reduction (10h 21min → 55 min): The combination of distributed tracing in Datadog APM, structured troubleshooting playbooks in OPE-002, and PagerDuty-managed escalation policies reduced average resolution time from 10 hours and 21 minutes to 55 minutes. Engineers can now correlate failures across services using trace data, identify root causes through APM dashboards and log analysis, and execute documented resolution procedures, replacing ad hoc, reactive firefighting with a repeatable, evidence-driven incident response workflow.

API Latency p95, 62% Reduction (25ms → 9.5ms): Optimization of Kubernetes scaling parameters through HPA and Karpenter tuning, combined with ElastiCache Valkey cache layer optimization, reduced p95 API response time from 25ms to 9.5ms. This improvement directly benefits the real-time operational workflows of airline and airport clients, reducing latency in deicing coordination, vehicle tracking updates, and flight data exchanges with airport operational systems.

Beyond the measured KPIs, the engagement delivered the following operational outcomes:

Full-stack observability established: Platform went from infrastructure-only monitoring to end-to-end visibility, Datadog APM distributed tracing, structured log analysis, CloudWatch synthetic canaries for endpoint availability, and KPI-defined alerting thresholds across all critical services.

Alert fatigue eliminated: Miscalibrated alerting replaced with severity-differentiated, KPI-aligned monitors integrated with PagerDuty on-call schedules and escalation policies, ensuring every alert represents an actionable operational event.

Scaling reliability under peak load: HPA and Karpenter tuning ensures the platform absorbs traffic spikes, including winter weather demand surges, without latency degradation or delayed resource provisioning.

Identity governance centralized: IAM Identity Center SSO with role-based permission sets deployed across all accounts, eliminating over-permissioned roles and establishing full auditability of access organization-wide.

Security posture hardened: GuardDuty EKS runtime monitoring, Security Lake centralized log aggregation, Amazon Inspector continuous vulnerability scanning, and KMS CMK encryption enforced across all data stores.

Disaster recovery validated: Pilot Light DR in us-east-2 with Aurora Global Database cross-region replication, targeting RTO of 4 hours and near-zero RPO, with DR infrastructure maintained in Terraform for full production parity.

Operational runbook formalized: OPE-002 documents structured daily, weekly, and monthly operational tasks alongside troubleshooting playbooks for every KPI-defined failure scenario, giving the team a consistent response framework.

Business Impact

The operational improvements delivered through this engagement translate directly into the company's ability to fulfill its commitments to some of the world's most demanding aviation clients, and to grow that client base with confidence.

Aircraft deicing is not a discretionary workflow. When winter weather activates deicing operations at a major hub airport, the Icelink platform becomes a real-time coordination system for dozens of vehicles, crews, and aircraft movements operating against precise departure schedules. In that context, a 10-hour MTTR is not an infrastructure metric, it is a potential multi-hour disruption to ground safety coordination at airports serving hundreds of flights per day. The 91% reduction in MTTR, from 10 hours and 21 minutes to 55 minutes, fundamentally changes the risk profile of a platform outage: what was previously an extended operational failure is now a contained, rapidly resolved incident.

The 85% reduction in MTTD is equally significant. At 20 minutes of average detection time, degradation in the Icelink platform could propagate through an active deicing operation before the engineering team was even aware of it. At 3 minutes, issues are surfaced before they become visible to airline dispatchers, ground crews, or pilots. For clients that include 3 of the 4 largest US airlines, operating under strict ground time commitments and regulatory reporting requirements, this shift from reactive to proactive operations is the difference between an internal engineering event and a customer-facing incident.

The 62% reduction in API latency (25ms → 9.5ms) directly improves the real-time responsiveness of the deicing coordination workflow: faster data exchanges with airport operational systems, more responsive vehicle tracking updates, and lower latency for the pilot-facing Hold Over Time application, all of which contribute to tighter, more reliable ground operations for the company's clients.

Perhaps most significant for the company's growth trajectory is what the engagement established structurally. With centralized identity governance, a formalized incident response framework, a validated DR strategy, and a full-stack observability model in place, the company can now onboard new airline and airport clients against a known, documented operational baseline, not against an ad hoc infrastructure that scaled faster than its governance model. For a company serving safety-critical aviation operations across multiple continents, that foundation is not a technical nicety; it is a prerequisite for sustainable commercial growth.

Get in touch

Take your business to the next level.