$14$15🟡 P3: Replica count too low (0 of 3) on agentic-workflow-backend / deployment / agentic-workflow
📅 Date: 2025-08-26 14:32 UTC
📊 State: FIRING
👤 Owner: Backend Team
🌍 Location: AWS:kauz-prod:Robert
📝 Description:
Grafana detected that the deployment agentic-workflow-backend
has insufficient replicas running in Robert cluster.
Condition was breached: 0 replicas available vs 3 expected,
sustained for more than 5m.
⚠️ Impact:
The agentic-workflow-backend is not serving traffic, causing request
failures in the agentic-workflow project. Users depending on this
backend may face service unavailability.
🔧 Next step: Check rollout with `kubectl describe deployment`
🔗 Links: [Dashboard] [Silence]
📡 Source: Grafana
--- Original ---
[FIRING:1] Deployment Failing Alert ...🟠 P2: Request volume too high (52k of 50k req/min) on alb-external-prod / loadbalancer / prod
📅 Date: 2025-08-26 13:15 UTC
📊 State: FIRING
👤 Owner: Infra Team
🌍 Location: AWS:970404667933:prod
📝 Description:
CloudWatch detected a sudden surge in request count on the production ALB.
Condition was breached: 52,000 requests/min vs 50,000 allowed,
observed in 4 of 5 checks over 10m.
⚠️ Impact:
The load balancer is near or above designed capacity.
If backend scaling does not keep up, users may experience
slower responses or request failures.
🔧 Next step: Validate backend auto-scaling and check error rates
🔗 Links: [CloudWatch Chart]
📡 Source: AWS CloudWatch
--- Original ---
❌ ALARM: "Performance Threshold Alert for Prod ALB (excl_5xx_4xx)" ...🔵 P4: SSL certificate renewed successfully (90 of 90 days) on managed-kauz.net / ssl-certificate / prod
📅 Date: 2025-08-25 13:02 UTC
📊 State: RESOLVED
👤 Owner: Infra Team
🌍 Location: AWS:kauz-prod:prod
📝 Description:
Updown.io detected that the SSL certificate for managed-kauz.net
was renewed.
Condition checked: new cert validity 90 of 90 days,
old cert replaced on all endpoints.
⚠️ Impact:
No user disruption. Secure connections remain valid.
If duplicate notifications appear, some servers may still serve the old cert.
🔧 Next step: Verify all nodes present the new certificate
🔗 Links: [Updown Panel]
📡 Source: Updown.io
--- Original ---
SSL certificate renewed for wasserhygiene.managed-kauz.net ...🔴 P1: Service failing with 504s (5 of 5 checks) on sandbox-barcode / url-check / sandbox-services
📅 Date: 2025-08-26 09:47 UTC
📊 State: FIRING
👤 Owner: Sandbox Team
🌍 Location: AWS:sandbox-services:sandbox
📝 Description:
Betterstack detected repeated 504 Gateway Timeout responses
from sandbox-barcode.
Condition observed: 5 of 5 health checks returned 504 errors
over 5m.
⚠️ Impact:
Sandbox BarCode service is fully unavailable.
All users accessing this endpoint are affected.
🔧 Next step: Check backend availability and restart the service if required
🔗 Links: [Acknowledge]
📡 Source: Betterstack
--- Original ---
Monitor: Sandbox - BarCode | Cause: Status 504 ...🟠 P2: Error rate too high (1.2 of 1.0 errors/sec) on lineup-service / endpoint:/lineup / spm-ingress-alerts
📅 Date: 2025-08-26 12:58 UTC
📊 State: FIRING
👤 Owner: Das Meta On-Call Team
🌍 Location: AWS:spm-ingress-alerts:prod
📝 Description:
Grafana alert detected that the 5xx error rate on lineup-service
endpoint:/lineup exceeded the defined threshold.
Condition: 1.2 errors/sec vs 1.0 allowed,
3 of 5 checks over 5m.
⚠️ Impact:
~10% of user requests to lineup-service are failing,
causing partial production disruption.
🔧 Next step: Check ingress logs and service backend for failures
🔗 Links: [Dashboard] [Silence]
📡 Source: Grafana via OpsGenie
--- Original ---
[FIRING:1] High 5xx Error Rate: Lineup Endpoint ...🟠 P2: Unexpected TypeErrors detected (62 of 0 expected) on yii2-website / controller:DefaultController.php / web-prod
📅 Date: 2025-08-26 03:27 UTC
📊 State: FIRING
👤 Owner: Suggested Assignee → Philipp R.
🌍 Location: AWS:web-prod:prod
📝 Description:
Sentry captured multiple TypeError exceptions in yii2-website.
Condition: 62 events observed vs 0 expected in the last 10m.
Code attempted to assign a string to MembershipTeamForm::$interval (int expected).
⚠️ Impact:
Pricing updates may fail for all users of the site.
Errors are ongoing and could affect subscription renewals
and membership management.
🔧 Next step: Fix type mismatch in MembershipTeamForm::$interval
🔗 Links: [Sentry Issue] [Repo]
📡 Source: Sentry
--- Original ---
TypeError /var/www/html/common/modules/premium/src/controllers/DefaultController.php ...