A full queue generally means that the agents collect more data than APM server is able to process. This might happen when APM Server is not configured properly for the size of your Elasticsearch cluster, or because your Elasticsearch cluster is underpowered or not configured properly for the given workload.
The queue can also fill up if Elasticsearch runs out of disk space.
So we need to tune up a bit apm-server config. I’m running my apm in kubernetes, for this we need to open apm-server.yml configmap and add the strings below:
queue.mem.events: 30000 output.elasticsearch.bulk_max_size: 20000 max_event_size: 1457600 max_header_size: 485760 idle_timeout: 90s read_timeout: 300s write_timeout: 300s output.elasticsearch.worker: 5 output.elasticsearch.timeout: 300
The main strings here are queue.mem.events and bulk_max_size, the default values are too low comparative to these. If you use it in production, consider increasing them.
- Internal queue size (queue.mem.events)
- Elasticsearch bulk size (output.elasticsearch.bulk_max_size).