Currently we use New Relic; as I’ve been exploring DataDog for metics and logs (being able to cross-reference them is neat), I also began to wonder whether anyone had successfully used DataDog’s APM tracer in Aptible’s environment before. As it is not as plug-and-play as just installing a gem, I’m tempted to conclude it’s not worth it, as it seems as if I will have to run a separate docker container to run the agent.
Thoughts/experiences welcome.
We’re in same boat, we use New Relic APM. Since we’d also like metric drains in data dog, would prefer to just have a single solution. Would love to hear other people’s experiences here.
Hi Jessica and Swaraj,
Which application framework(s) are you each looking to instrument with Datadog APM? For many of these frameworks, it appears they’ve simplified setup to make it indeed nearly as plug-and-play as adding a gem (and a small configuration block). See their current APM quickstart instructions for Rails apps, for example: https://docs.datadoghq.com/tracing/setup/ruby/#quickstart-for-rails-applications
— Frank
Hi Frank,
I’ve got metrics (push-button) and now logs (as events, via a logstash container). I looked at the APM stuff and while the tracer is just a gem, the tracer doesn’t hit DD directly via that code–it uses the DD agent, which I couldn’t see running on Aptible (or at least it was too big a lift for me) because the agent, even the dockerized version, appears to rely on the traditional docker host-container model, where you are doing the installation of the agent on the docker host where you know all your other containers are running. Maybe I’m missing something here, but that doesn’t sound like Enclave as I understand it…
Jessica
I’ve managed to get the agent running as a separate process from my procfile using supervisord, but similar to @jessica mentioned, it seems like the gem can find the agent. I’m trying to run it directly from within the image and pushing it to the background, so I’ll see if that works.
So this is the error I get when I try to run a trace now:
irb(main):007:0> E, [2018-04-27T20:33:06.084109 #1] ERROR -- ddtrace: [ddtrace] (/home/app/athene/vendor/bundle/ruby/2.3.0/gems/ddtrace-0.11.4/lib/ddtrace/transport.rb:112:in `rescue in post') Failed to open TCP connection to localhost:8126 (Connection refused - connect(2) for "localhost" port 8126)
Brief update…
I’m running the DD agent now in its own container–DD told me that not mounting host volumes was not a big deal, so I decided to try it. It seems to function, i.e., it can communicate with DD’s API.
I’ve installed the ddtrace gem, attempting to relay the tracer gem’s output to it via an HTTPS endpoint. This doesn’t appear to be working the way it should, yet, but I’m working with DD’s support team on it.
@Chase, I don’t think localhost is going to be sufficient to find where the agent is listening, given Aptible’s architecture. Although what would you tell it instead, for your setup, I don’t know. I can at least tell you where you tell it. This is my config file:
# config/initializers/datadog-tracer.rb
Datadog.configure do |c|
c.use :rails, service_name: 'qa-on-aptible'
c.tracer hostname: 'my-endpoint.on-aptible.com', debug: true
end
More soon.
IT WORKS!
Really, in hindsight, the setup was easy: all it requires is a TCP endpoint, and voilà, it finds the Datadog agent in its separate container and sends along the data.
Basic instructions (if you want more detail, just ask):
Create an app for the agent. I set up mine in a separate environment.
Set these ENV variables (your situation may differ):
DD_API_KEY=your-key-here
DD_APM_ENABLED=true
aptible deploy --app <name-of-your-app> --docker-image datadog/agent
aptible endpoints:tcp:create --app <your-app-name> cmd
5a) Add the tracer to your application. For Ruby, I added the ddtrace gem.
5b) Configure the tracer. Our Ruby config is:
# config/initializers/datadog-tracer.rb
options = {
service_name: 'up-to-you',
env: Rails.env
}
Datadog.configure do |c|
c.tracer hostname: 'elb-welld-00000.aptible.in' #replace with your hostname
c.use :rails, options
end
The first time you run it, it’s helpful to have debug-level logs, so change that c.tracer line to:
c.tracer hostname: 'elb-welld-00000.aptible.in', debug: true
NOTE: Datadog does not offer BAAs right now; I set this up on our qa server.
For the sake of clarity (it looks like there has been some confusion across customers reading this thread):
The instructions Jessica provided here are to set up Datadog APM (i.e. application-level metrics), they’re not required to set up a Metric Drain (i.e. container-level metrics).
To use a Metric Drain with Datadog, just go to the Metric Drains tab in the Dashboard, click on “Create Metric Drain”, and select the Datadog option.
Hello fearless Enclave denizens!
I’ve got more good news–with a simple modification to the docker image of the DataDog agent, it can collect Enclave logs! All you need is a TLS endpoint in addition to the regular TCP one (since Log Drains only send to TLS-TCP, and the agent listens for plaintext). I have used the agent’s log importer to replace the logstash-to-datadog setup I (sadly) spent a LOT OF TIME ON before. Such is the way it goes…
So now we have APM, logs, and metrics in ONE PLACE: DataDog.
NOTE: Again, DataDog has not yet finalized its promised BAA offering, so beware of porting PHI!
I’ll post a how-to soon, but if you have urgent queries, feel free to respond here.
Jessica
Jessica Hello. do you have any updates as i see this was written a long time ago and i am wondering about either DataDog has finalized their promises. lately i am on https://suppsforlife.to/category/peptides/ and that’s why i don’t keep track of everything. thanks a lot for sharing.
I haven’t heard a word from them about it, no… I don’t think they are offering it yet.
Thanks a lot, @jessica for sharing how you got the APM working in Aptible, very useful!
Not sure if it’s only me, but I had to add one more ENV variable to the datadog-agent aptible app in order to get my setup working: DD_APM_NON_LOCAL_TRAFFIC=true - just in case anyone else will run into the same issues I did, try this. Without that, I was getting an error similar to: ERROR:ddtrace.writer:cannot send spans to elb-xxxxxxx-XXXX.aptible.in:8126:
+1. Thanks a ton, @jessica for posting your notes in here. Was super helpful.
Something that I needed to do in addition is set this environment variable DD_BIND_HOST=0.0.0.0 to allow the Datadog Agent to accept traffic from non-local host.
Without this setting, the trace was being sent to the Datadog Agent but was being rejected and I was receiving a EOF Reached error on our app.
Kudos to @ashley for helping me in debugging this issue 