This reference architecture outlines a scalable and developer-friendly background job processing system using Laravel, Horizon, Kubernetes, KEDA, Redis/RabbitMQ, and Prometheus/Grafana.

It is designed to:

• Simplify developer onboarding and local testing

• Support autoscaling across multiple priority queues

• Provide clear observability and operational safety

• Ensure queue workloads are isolated, recoverable, and easy to debug

This setup is ideal for Laravel-based teams who want predictable background job execution and elastic scaling without introducing heavy orchestration or complex deployment pipelines. It enables modern infrastructure practices while staying approachable to small and mid-sized teams.

This architecture is driven by a specific set of functional and non-functional requirements:

• Support for Laravel-based background jobs

• Developers must be able to submit jobs into priority queues

• A web UI is needed for testing job dispatch

• Ability to run locally for fast iteration

• Must work seamlessly in production with minimal configuration changes

• Horizontal scalability of job execution based on demand

• High observability: visibility into job status, queue depth, failure rates

• Fault tolerance: restart failed workers, prevent stuck queues

• Secure by default: no unnecessary external exposure

• Simple to extend for new queues or jobs

These requirements prioritize a balance between developer productivity and production-grade reliability.

This visual outlines how the components interact: a dedicated monitoring pod runs the Horizon UI, while independent worker pods are autoscaled by queue length using KEDA. Redis or RabbitMQ acts as the queue backend, and metrics flow into Prometheus for visibility.

This is a dedicated pod running php artisan horizon in monitoring mode. It:

• Provides a unified UI for viewing job and queue status

• Connects to the shared Redis instance to aggregate data from all workers

• Does not execute any jobs itself

This pod is the central interface for developers and operators to understand system state without exposing sensitive execution pods. It enables safe inspection of queue behavior and job failures across the environment.

Each pod runs Horizon to manage workers via Laravel's built-in supervisor logic. Key traits:

• Runs a single queue type (e.g. high, default, low) for clarity and isolation

• Configured via static horizon.php supervisor definitions

• Exposes liveness and readiness endpoints to Kubernetes

• Horizontally scaled by KEDA using queue-specific metrics

This pattern ensures that each type of workload is independently scalable and observable. Having separate deployments per queue avoids contention and simplifies resource tuning.

Acts as the queue backend. Redis is preferred for full Horizon observability. RabbitMQ may be used but requires external monitoring and does not integrate with Horizon's metrics.

All job metadata and execution status are stored here, which enables job tracking, retries, and monitoring. The architecture supports RabbitMQ if needed, but Redis remains the default for simplicity and native Horizon support.

Provides infrastructure-wide visibility beyond Horizon. You get:

• Queue depth tracking (especially critical for RabbitMQ)

• Pod-level metrics for resource usage

• Alerts on stuck queues, unconsumed jobs, or unhealthy pods

This decouples operational alerting from application logic and gives infrastructure teams full visibility without depending on the Laravel runtime.

A developer-facing Laravel UI that:

• Sends jobs to the queues for testing or production

• Contains example code for workers and job dispatch

• Can be integrated into business apps directly

This ensures teams can test the architecture without writing boilerplate and helps standardize job dispatch patterns.

Using priority-based queues like high, default, and low lets developers separate critical jobs from batch/slow tasks. Each queue is handled by its own worker pod group. This avoids having long-running or expensive jobs block lightweight tasks.

Developers dispatch jobs as follows:

This allows infrastructure teams to assign scaling policies, resource budgets, and monitoring to each queue type independently. It also introduces a predictable pattern for queue growth and capacity planning.

KEDA enables scaling Horizon worker pods based on live queue depth. Configuration is queue-specific:

• minReplicaCount ensures warm pods for latency-sensitive queues

• cooldownPeriod avoids flapping

• Custom metrics adapters can use Redis or RabbitMQ metrics

Autoscaling ensures that low-traffic queues consume minimal resources, while bursty or critical queues can expand capacity in real-time.

Workers can crash, hang, or fail to connect to Redis. This setup mitigates those issues with:

• Startup probes that check Redis connectivity before allowing pod readiness

• Liveness probes that restart pods if Horizon hangs

• Readiness probes to prevent traffic to non-ready pods

Laravel's built-in SIGTERM handling ensures graceful shutdown without losing in-flight jobs. This leads to fewer retries and better job durability.

Only the central Horizon UI is exposed externally, secured via VPN and SSO. Worker pods:

• Have no external ingress

• Are accessible only within the cluster (for health/debugging)

• Share a secured Redis instance

This architecture ensures sensitive job internals, job payloads, and operational controls are never exposed unintentionally.

Horizon shows basic job and queue stats, but infrastructure-level monitoring is handled via Prometheus:

• Alerts when queue depth exceeds thresholds

• Visibility into pod CPU/mem usage

• Tracking of message age to catch unconsumed queues

If RabbitMQ is used instead of Redis, metrics must be collected using RabbitMQ exporters. In either case, Grafana can visualize overall queue health, trends, and alert thresholds.

The boilerplate GitHub repo includes:

• Local docker-compose for testing: Laravel, Redis, Horizon, Web UI

• Kubernetes manifests for deploying Horizon UI and worker groups

• Example job handlers by priority (mapped to queues)

• A job dispatch UI web app ready to be extended

To get started, developers:

1. Clone the repo

2. Implement their job logic in app/Jobs/

3. Use docker-compose up locally or deploy to Kubernetes

This workflow ensures that developers can iterate quickly and deploy confidently, using the same job patterns in local and production environments.

• Adding a queue: Define in config/horizon.php, add new deployment with the right KEDA scaler

• Adding a job: Create a Job class and dispatch to the target queue

• Debugging: Use Horizon UI for job state, and Grafana for queue-level metrics and alerts

This modularity enables team scaling and specialization. Infra teams manage resources and autoscaling while developers focus on logic.

• Horizon only supports one queue driver (Redis). For RabbitMQ, monitoring must be external.

• Workers must run php artisan horizon — queue:work processes are invisible to Horizon.

• Queue backlog due to missing handlers will not error by default but will be caught via Prometheus alerts

• Redis must be scaled appropriately if many Horizon instances are writing concurrently

These tradeoffs simplify the architecture but assume queue discipline and observability are in place. This is an intentional trade for simplicity, performance, and Laravel-native tooling.

This architecture balances developer ergonomics, observability, and scalability. It is easy to adopt, secure to run in production, and extensible for complex queueing needs.

By combining Laravel Horizon with modern Kubernetes patterns and an observability-first mindset, teams can confidently ship background jobs and scale them as their product grows.