I’ve recently updated the container I had running to use Datadog Agent v6.12.1, and in so doing, I was able to document how the process works. I’m posting this in case it helps others.
ALSO, DataDog now signs BAAs! Let those logs rip!
NOTE: This information does not apply to container metrics! Metrics are set up through a push-button metric drain that can drain to DataDog without a container!
The dockerized DataDog Agent, under normal (that is, non Aptible Deploy) circumstances, attaches to the docker daemon via the unix socket. In Aptible, this isn’t possible, and so the standard features of the Agent won’t work.
That said, you can still get trace data and logs forwarded by the agent via TCP forwarding from various containers/log drains: the trace and log collection agents listen on specific ports and then forwards them to DataDog’s API.
The agent lives in a container with two endpoints, a tcp endpoint and a tls endpoint. The TLS endpoint is necessary because Deploy log drains only send to TCP/TLS, whereas the Agent is listening in plaintext.
The trace agent requires configuration within your application itself to send the traces. In our case, since we have a ruby app, we use a gem to send these traces to the TCP endpoint. (You can see my config in an earlier post.)
Log collection from other Deploy containers requires the setup of a custom collection agent.
Building the Docker container, part one
Because we need to configure this custom collection agent, we need to build our own Docker image.
- Clone this repo
- Download the
.debdatadog-agent package for the build. (Check your versions, architectures, etc.)
- cd to the directory
Dockerfiles/agentand move the binary package to that location.
- It’s probably not a bad idea to attempt to build it just as is, to see that it does build.
- Building the Docker container, part two
Now that we’ve got the potential to build a container, we can make our modifications
- You need to add port 10518 to the
- You need to
COPYin the relevant configuration file for the custom log collection agent
The documentation on this isn’t terribly clear. It describes putting the same configuration file in two places. Based on the log output, I suspect that it is only needed as a
conf.d, but I did put it in both places and I haven’t bothered to check if it works without both of them.
Here are the two lines I added to the
COPY logs.conf.yaml /etc/datadog-agent/welldapp.d/conf.yaml COPY logs.conf.yaml /etc/datadog-agent/conf.d/aptible.d/conf.yaml
And here is the collection file that I placed in those locations:
init_config: instances: logs: - type: tcp port: 10518 service: welldapp source: aptible
Note how the
source names match the names of the destination folders in which they are placed in the Docker image.
- Configuring the Agent via ENV variables
I was able to achieve all the configuration I needed using ENV variables.
APTIBLE_PRIVATE_REGISTRY_USERNAME=<removed> APTIBLE_PRIVATE_REGISTRY_PASSWORD=<removed> DD_API_KEY=<removed> DD_APM_ENABLED=true DD_APM_ENV=enclave DD_APM_NON_LOCAL_TRAFFIC=true DD_HOSTNAME=dd-container-on-enclave DD_LOG_LEVEL=off DD_LOGS_CONFIG_LOGS_DD_URL=tcp-encrypted-intake.logs.datadoghq.com:10516 DD_LOGS_ENABLED=true DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=false DD_PROCESS_AGENT_ENABLED=disabled
DD_APM_ENVvariable is new since 6.1.1; previously all my metrics were env:none, so it’s nice to be able to set it to something.
- Note that we use the URL for encrypted log intake because we have a BAA.
- I have set the last two variable to false since, without a properly connected Agent, that functionality isn’t available anyway. (They are likely false by default, but I wanted to be clear.)
- Configuring the endpoints
The agent’s container needs two endpoints.
- TCP endpoint. This specifies BOTH ports that the Agent is listening for traffic on. In my case, that’s 8126 and 10518.
- TLS endpoint. This specifies only the port that the log drain will be connected to. In my case, that’s 10518.
- You should set these up using the CLI
- Set up log drains
Each of the aptible environments from which you want DataDog to collect logs. The log drain must point at the TLS endpoint.
Questions? Feel free to ask.
And if there is anyone out there proficient in the right code base, I’d love to work with you on customizing the DataDog agent itself to have an Aptible Deploy mode built in so that we could override the expectation of the Docker socket connection and thus get rid of the constant error stream that this lack of connection generates. There are probably enough DataDog/Aptible users that they’d be willing to consider a pull request.