UDP Communication between containers

Use Cases
1

Is there a way to allow UDP communication between containers or any workaround for it? We need to emit metrics to our agent via StatsD

2

The AWS LoadBalancers that back Endpoints only support TCP so services running on Aptible cannot receive data via UDP. Your options are:

  • Host the agent in your own AWS VPC and set up VPC peering with your Stacks so that they can send the metrics to the agent over the private network. I believe this is what most users have done if they decide to use the StatsD agent.
  • Run the agent in each container you want to collect metrics for and configure your services to send the data to the agent running in the same container. Connecting to services running within the same container doesn’t require an Endpoint so they can use UDP.
  • Create a custom service that runs the agent and accepts the metrics over TCP before forwarding them to the agent over UDP. There was a user that attempted to set up a service that accepted metrics InfluxDB format, likely using telegraf, then forward them on to the agent but I don’t know if they were successful or not.

I’m not sure if this is the same but the Datadog application performance monitor (APM) agent supports tracing application metrics so if you’re already using the APM in your containers, or would like to, this may be a good option.

3

Following this - would be interested on any guidance for setting up the datadog APM agent in existing container - currently trying to get this working alongside the dd-trace-rb ruby gem from datadog

4

Hi @abramf , Thanks for asking the question here and I apologize for not getting to it sooner. I’m going to share @MichaelW 's answer here as well:

The gem is trying to open a TCP connection using the default port and host IP route. You will need to create a TCP Endpoint and then set the tracer hostname to that of the Endpoint.

We cannot provide very much help for 3rd-party services like DataDog, but we’ve seen the DataDog agent be successfully implemented as follows:

  • Run the Datadog Agent as it’s own application, using the prebuilt image from Docker Hub : aptible deploy --app datadog-agent --docker-image=datadog/agent

  • Add three important pieces of configuration to the agent App : aptible config:set --app datadog-agent DD_APM_ENABLED=true DD_APM_NON_LOCAL_TRAFFIC=true DD_API_KEY=your_key_here (see the Datadog documentation for these and other settings)

  • Create an internal TCP Endpoint to allow your instrumented applications to publish APM information to the DD Agent: aptible endpoints:tcp:create --app datadog-agent cmd --default-domain --internal

  • Instrument your Ruby App as documented here: https://docs.datadoghq.com/tracing/setup_overview/setup/ruby/

  • Set the tracer option hostname to that of the above Endpoint : https://docs.datadoghq.com/tracing/setup_overview/setup/ruby/

Then, on your Apps that send data to the agent, you only have to set a couple of environment variables

DD_AGENT_HOST=app-NNNNN.on-aptible.com (the endpoint URL from the aptible endpoints:tcp:create command above)
DD_TRACE_ANALYTICS_ENABLED=true

Hope this information helps!

5

We were able to get this working with your help @JoshJ and a few tweaks.
Currently trying to figure out how to capture ruby runtime metrics (especially as related to mem / Ruby GC) via StatsD to the agent without:

  • installing the agent alongside the application (seems sketchy)
  • using VPC peering (many security considerations to work through)

Seems like the only option would be to send the metrics over TCP (although the gem likely does not support this) and then forwarding them on to the agent. Will try to update if I find a way to do this.

6

Regardless of if your StatsD client supports TCP, I don’t think the DogStatsD server that runs with the agent supports TCP which limits your options to:

  • Run the agent in the same container.
  • Host the agent outside of Aptible and use a network integration to communicate with the agent (e.g. VPC peering).
  • Use a custom agent/server that supports TCP for DogStatsD.
  • Send the data directly to DataDog instead of using the agent (not sure how feasible this actually is).
  • DataDog has a StatsD backend that you may be able to use with a standard StatsD client but it may not support all of the features that the DogStatsD server and client do.

If you haven’t already I recommend reaching out to DataDog to see if they have any recommendations or you could open an issue/feature request on the datadog-agent repo for DogStatsD TCP support. Based on this comment it seems they have some ideas for workarounds.