Stop filebeats send old data to log stash

T16:31:49.391+0100ДћRROR logstash/async.go:256Дџailed to publish events caused by: client is not connected T16:31:19.305+0100ДћRROR pipeline/output.go:121Дџailed to publish events: client is not connected T16:31:17.967+0100ДћRROR logstash/async.go:256Дџailed to publish events caused by: client is not connected But the filebeat instance which remained unrestarted still doesn't push logs and logs things like (random snippet from log taken more than 24 hours since the problem was resolved):

This is all OK, logstash was massacring CPU and likely was unable to keep up with connections.Д«ut then, the problem was resolved early next day, logstash was restarted, those filebeats which were restarted work since then happily. T19:46:50.556+0100 ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected T19:46:18.110+0100 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://:5044)) with 5 reconnect attempt(s) T19:44:47.796+0100 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://:5044)) with 2 reconnect attempt(s) T19:44:40.506+0100 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://:5044)) with 1 reconnect attempt(s) T19:44:36.909+0100 ERROR pipeline/output.go:121 Failed to publish events: client is not connected T19:44:35.025+0100 ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected

Restarting filebeat helped, but there is sth wrong in the fact that it didn't manage to recover itself (after all, in normal logstash restart/temporary inaccessibility case I never faced similar problems)

Even long after logstash was restarted and the problem it faced resolved, running filebeat instance never recovered (the instance I forgot to restart was still disconnected more than 24 hours after the problem was resolved). вЂ¦ but for some reason filebeat remained in this state forever. Sooner or later filebeat started to report timeouts (properly, logstash didn't manage to handle communication fast enought) Logstash (to which filebeat's connect) faced big performance problems (it was due to inefficient XML parsing, elastic/logstash#11599, but this is only context of this issue, if I were to test this behaviour on purpose I'd probably write some busy loop in ruby script triggered from a filter) Somewhat strange situation, but I observed it on many machines (filebeat 7.5.2, logstash also 7.5.2): I am not sure whether this is sufficient for detailed analysis, but I decided to report the problem so sb could take a look at filebeat behaviour having this context in mind.

Stop filebeats send old data to log stash