Continuous Failed to make HTTP(S) request Warnings

[twcchu]

Question

We are getting warnings for Failed to make HTTP(S) request for the datadog_metrics sink every few hours. The buffer is not discarding events or filling up according to the metrics. We are using default sink buffer and batch settings. Are there other metrics we can look at to triage this warning or other settings that will keep the connection from resetting?

Vector Config

sources:
  vector_metrics_filebeat:
    type: internal_metrics
    scrape_interval_secs: 10
sinks:
  datadog_metrics_filebeat:
    type: datadog_metrics
    inputs:
      - metrics_fb_tag
      - logs_count
    default_api_key: ${DATADOG_API_KEY}
    site: datadoghq.com

Vector Logs

{"timestamp":"2025-11-12T15:58:46.820474Z","level":"WARN","message":"HTTP error.","error":"connection error: Connection reset by peer (os error 104)","error_type":"request_failed","stage":"processing","target":"vector::internal_events::http_client","span":{"name":"http"},"spans":[{"component_id":"datadog_metrics_filebeat","component_kind":"sink","component_type":"datadog_metrics","name":"sink"},{"request_id":310843,"name":"request"},{"name":"http"}]}
{"timestamp":"2025-11-12T15:58:46.820541Z","level":"WARN","message":"Retrying after error.","error":"Failed to make HTTP(S) request: Failed to make HTTP(S) request: connection error: Connection reset by peer (os error 104)","internal_log_rate_limit":true,"target":"vector::sinks::util::retries","span":{"request_id":310843,"name":"request"},"spans":[{"component_id":"datadog_metrics_filebeat","component_kind":"sink","component_type":"datadog_metrics","name":"sink"},{"request_id":310843,"name":"request"}]}
{"timestamp":"2025-11-12T18:23:03.881544Z","level":"WARN","message":"HTTP error.","error":"connection error: Connection reset by peer (os error 104)","error_type":"request_failed","stage":"processing","target":"vector::internal_events::http_client","span":{"name":"http"},"spans":[{"component_id":"datadog_metrics_filebeat","component_kind":"sink","component_type":"datadog_metrics","name":"sink"},{"request_id":315970,"name":"request"},{"name":"http"}]}
{"timestamp":"2025-11-12T18:23:03.881628Z","level":"WARN","message":"Retrying after error.","error":"Failed to make HTTP(S) request: Failed to make HTTP(S) request: connection error: Connection reset by peer (os error 104)","internal_log_rate_limit":true,"target":"vector::sinks::util::retries","span":{"request_id":315970,"name":"request"},"spans":[{"component_id":"datadog_metrics_filebeat","component_kind":"sink","component_type":"datadog_metrics","name":"sink"},{"request_id":315970,"name":"request"}]}

[thomasqueirozb]

When running Vector in debug mode you are likely to see Sending HTTP request. logs.

vector/src/internal_events/http_client.rs

Lines 31 to 44 in 5c16191

	impl<T: HttpBody> InternalEvent for AboutToSendHttpRequest<'_, T> {
	fn emit(self) {
	debug!(
	message = "Sending HTTP request.",
	uri = %self.request.uri(),
	method = %self.request.method(),
	version = ?self.request.version(),
	headers = ?remove_sensitive(self.request.headers()),
	body = %FormatBody(self.request.body()),
	);
	counter!("http_client_requests_sent_total", "method" => self.request.method().to_string())
	.increment(1);
	}
	}

For information on controlling log verbosity refer to the docs.

This will show you exactly what is being sent to datadog.

However, I don't think that will tell you anything useful. Connection reset by peer (os error 104) usually indicates a connection error originating from the server, and there is little that can be done. (From what I read online it could also be that your system time is incorrect or maybe even a firewall issue but those would need more investigation. Again, it could be that datadog itself is resetting the connection).

FYI: Under the hood we're using hypers http client. Maybe tuning Vector's verbosity to log hyper's trace/debug logs will show you more information.

[pront]

Also, some source metrics to watch out for:

• http_client_errors_total and look for the error_kind tag
• http_client_requests_sent_total
• http_client_responses_total and you can use the status tag to filter for successful vs failed requests

For the Datadog metrics/logs sink connection issues, filter with component_type="datadog_metrics":

• component_sent_events_total - Should keep increasing (data is flowing)
• component_errors_total - zero or very low (retries are not reflected here)
• component_discarded_events_total - permanent failures will show up here

If the above feels overwhelming I would start with:
component_errors_total and component_discarded_events_total.

The following guide explains how to read the above metrics: https://vector.dev/guides/developer/debugging/