Skip to content

How to fix Elastic APM error (503): queue is full

A full queue generally means that the agents collect more data than APM server is able to process. This might happen when APM Server is not configured properly for the size of your Elasticsearch cluster, or because your Elasticsearch cluster is underpowered or not configured properly for the given workload.

The queue can also fill up if Elasticsearch runs out of disk space.

So we need to tune up a bit apm-server config. I’m running my apm in kubernetes, for this we need to open apm-server.yml configmap and add the strings below:

queue.mem.events: 30000 
output.elasticsearch.bulk_max_size: 20000
max_event_size: 1457600
max_header_size: 485760
idle_timeout: 90s
read_timeout: 300s
write_timeout: 300s
output.elasticsearch.worker: 5
output.elasticsearch.timeout: 300

The main strings here are queue.mem.events and bulk_max_size, the default values are too low comparative to these. If you use it in production, consider increasing them.

  • Internal queue size (queue.mem.events)
  • Elasticsearch bulk size (output.elasticsearch.bulk_max_size).
Published inKubernetesLinuxOther

Comments are closed.