Solution for logs AND metrics?

1

I love Aptible’s push-button integrations… I have tried SumoLogic for logs and DataDog for metrics. Both were really easy to set up. The question I have is whether Aptible can get both data types into either of those services, meaning: Is there a way to get metric data over to Sumo Logic? or Is there a way to get logs into DataDog? I’d love to be able to leverage one tool to look at both data streams.
Thanks,
Jessica

2

Currently, we don’t support those, but I’ve made a note that they were requested here.

That said, for logs in particular, one option is to run something like Logstash as an App on Aptible, and route your logs there (using an Aptible Log Drain, using e.g. syslog or HTTPS) for processing and routing. With this approach, you can route your logs wherever you’d like.

3

Thanks, Thomas. I did notice myself that DataDog supports Logstash as a source. But if I have separate environments for qa, demo, and prod, would I have to run 3 instances of Logstash to get each environment’s logs to DD?

4

Not necessarily. If you’re running your own app to process logs, you’re free to do whatever you want, including routing all your logs through a single instance of said app. The main (only?) caveat you should consider is that your app needs to be accessible from all your environments.

5

Just an update in case anyone is following. I have a logstash container now forwarding logs to Datadog. Hope to get a write-up of it this month, but if anyone is curious before then, feel free to ping me.

6

Hey @jessica and all,

I wonder what’s your experience with sending the logs to Datadog? Is everything working as expected or are there any performance issues? We are looking at a new solution for logging (we tried ES and LogDNA but neither of these are suitable for our needs, so looking at Datadog logs now, since they can sign BAA as well now).

Any feedback will be appreciated.

Thanks,

Petr

7

Hi Petr,
I have a dockerized container with the DataDog agent inside it that sends logs and also does the performance metrics/monitoring. I was a bit of work to understand it and get it going but it just works… and I’m happy to share my notes if you do decide to go this route.

I’ve been asking DD about a BAA for a while now–do you have written documentation that says they do now offer them? Since every time I ask, it turns out that it’s (still) not true… We’d love to have one ourselves to do more with our logs.

8

Hi Jessica,

I’m glad things works out for you with DD logs. We implemented LogDNA to gather logs from our ~25 Aptible environments and were very happy with it as it resolved many problems we had previously with ES in each environment around user access management, alerting on logs, creds rotation etc, but later we discovered missing logs and alerts not being sent for logs, so we are again on a lookout for a better logging solution.

We had a call with DD team last week and we told them we are very happy with them and we like most of their offerings, just would love to use also the logs but we cannot since they cannot sign BAA, and the DD people told us they can sign it now. So we should soon speak to the DD logging team and have an in-depth conversation about sending our aptible logs to DD. I believe if you would reach now to DD, they would tell you the same, that they can sign the BAA with you as well.

Lastly, if you could share any notes from your DD Logs project, that would be really appreciated.

Thanks!

9

Hi Petr,
Sorry for the delay.
It’s quite simple–or at least the notes I took at the end suggest it is so… Hope this helps!

  1. Create and configure your Aptible environment You may need or desire additional vars. (See this documentation.)
# some variables
DD_API_KEY=YOUR_VERY_LONG_KEY
DD_APM_ENABLED=true
  1. Start the agent in Aptible
    The agent container, dd-agent, is started directly off the pre-built image provided by DataDog:
aptible deploy --app <your-app-name> --docker-image datadog/agent
  1. Setup TCP endpoint
    The only additional Aptible setup is a TCP endpoint pointed at the agent’s port, 8126. (If I remember correctly, a TCP endpoint will give you at TLS endpoint as well, so if you see two endpoints afterwards, that’s why.)
aptible endpoints:tcp:create --app <your-app-name> cmd --ports 8126
# In fact, you do not need to list the port if it's only one; it will use the default.
  1. Configure your code to send application data to datadog. We are a ruby app, so we used and configured the ruby ddtrace gem. FWIW, here is the relevant config file.
# config/initializers/datadog-tracer.rb

def datadog_tracer_enabled?
  ENV['DATADOG_TRACER_ENABLED'] == 'true'
end

options = { 
  service_name: ENV['SERVER_ENV'], 
  env: Rails.env
}

Datadog.configure do |c| 
  c.tracer enabled: datadog_tracer_enabled?, hostname: 'your-tcp-endpoint.aptible.in'
  c.use :rails, options 
end
10

Hi Jessica and thanks a lot for you reply - just seeing it today.

We actually have a similar setup to collect the APM logs and it does work fine. What we are looking for now is implementing a solution to send all the other logs (the application / DB logs drained by the Aptible log drain) to datadog. For that it looks like we need to setup a logstash container for receiving and parsing/cleaning all the logs and then output them to DD logs for storage and alerting etc. I assume you are not sending the app/db logs to DD logs, correct?

11

Hi Petr,
Actually, that is the beauty of the datadog agent–it handles all of the communications between our application and datadog, including gathering the logs.
Turns out what I should have done before is refer you to this other thread: https://community.aptible.com/t/using-datadogs-apm/314/8; reading my old self, who clearly knew more than my current self, you add the TLS endpoint in addition to the regular TCP endpoint and THEN the agent can receive the logs.
The thing is, that post mentions “a simple modification” DataDog’s docker image. I’d have to go track down what that is for you, since I don’t know offhand.
I’ll see if I can find where I built that container… then you should be set.
Trust me, though,this is way better than a logstash-to-datadog setup, which I did try before I tried the agent.
Jessica

12

Sadly, I can’t find my Dockerfile, but according to docker history (TIL!), I’m pretty sure that the only change I made was to expose port 10518 (in addition to ports 8125/udp and 8126/tcp).