Datadog Unix Agent

A Datadog Agent specifically designed for Unix-based systems.

Note: This agent has been designed with portability in mind but has only been tested at the
time of this writing on AIX.
If you have any questions, please contact our support team.

The goal of this agent is to provide support for miscellaneous Unix operating systems not supported
by our currently available agents. To do so, it has been stripped of unnecessary bloat and cut around
less relevant use-cases given the target platforms. Only a subset of key and OS-specific features have
been implemented, please do not expect feature parity.

AIX

Omnibus Build

An omnibus build is available for the agent. The omnibus build provides a self-contained
environment, shipping all required dependencies, including python3 and other depending shared
objects. The omnibus build will however depend on a small subset of system-level dependencies
expected to be present on any AIX LPAR image; namely: libc, libpthreads, libdl and libcrypy.

Unfortunately, disparities exist between some of the system libs available symbols across AIX
technology levels on the same AIX version. This is an AIX particularity and means that we can
only guarantee the agent will run successfully on machines matching our builders' technology
level. That is:

Target Platforms

The omnibus build has been build and tested on the target platforms at the time of this writing:

• AIX 6.1 TL9 SP6
• AIX 7.1 TL5 SP3
• AIX 7.2 TL3 SP0

Should you attempt to install and run the agent package on a lesser version your mileage will
vary, you may or may not have all required symbols available. We are working to improve our
support across al target platforms.

Omnibus Bootstrap

To help with the setup of the omnibus builder we have a (still a bit flaky ;) script that should
help you get up and building in a jiffy or two. It's a basic shell script located in the omnibus/
folder that will help you install, amongst others:

• YUM (rpm package manager)
• Required build toolchain (gcc, libffi, ...)
• Ruby
• Required tools (sudo, GNU tar, bundler)
• Patch required libyajl-gem

Build Baseline Requirements

The omnibus build now ships all these dependencies, most notably openssl and python, all properly
linked and only requiring AIX baseline system level dependencies expected to be available:

• libc
• libpthreads
• libcrypt
• libdl

Build Runtime Requirements

As mentioned above all requirements are bundled with the omnibus installer. For more details take
a look at the omnibus/ directory to take a look at the implementation.

Installation

Download links for the latest releases can be found on this page.

The installer may be executed as follows (as root):

installp -aXYgd ./datadog-unix-agent-<version>.powerpc.bff -e dd-aix-install.log datadog-unix-agent

This will install the agent in /opt/datadog-agent.

Note how we're logging to dd-aix-install.log, you may skip that by removing the -e switch.

Running the agent

The expected configuration location is:

/etc/datadog-agent/datadog.yaml

That said, config file will be searched in this order (with the first match being
taken):

• /etc/datadog-agent/datadog.yaml
• ./etc/datadog-agent/datadog.yaml
• ./datadog.yaml

This should support legacy configuration locations from early adopters who may have
installed earlier dev images. Please try to update to the preferred location to avoid
issues in the future with potential deprecations.

A sample configuration file may be found in /etc/datadog-agent/datadog.yaml.example.

A basic configuration will typically require your datadog API key. Should you require
to submit your metrics to the EU instance, the site configuration option is available.
You may also override the dd_url manually, but that should not be required.

Occassionally a proxy configuration must be specified depending on your network setup.

In versions >=0.7.0, an SRC subsystem is created so you can now manage the agent
using the usual AIX commands to manage it. Thus, with the configuration in place,
you may start the agent with:

startsrc -s datadog-agent

You may check the status with:

lssrc -s datadog-agent

And you may stop it with:

stopsrc -s datadog-agent

Please note that an inittab entry is now created automatically on package installation
and the agent will be brought up automatically when runlevel 2 is entered. If you wish
to disable this behavior please run the corresponding rmitab command:

rmitab "datadog-agent"

The default inittab configuration enables the service in respawn mode, such that if
the agent should crash, it will be automatically respawned. The goal of this behavior
is to help avoid a loss in observability in the event of an agent crash. AIX service
management is a little rudimentary, so if this behavior is too invasive for your liking
you may change the behavior from respawn to once:

chitab "datadog-agent:2:once:/usr/bin/startsrc -s datadog-agent"

Note: in this scenario the agent will not be automatically restarted in the event of
a crash.

Deprecated

The following instructions refer only to earlier iterations of this project and do not
apply to the stable agent (1.0.0) or dev versions >=0.7.0. These are here for historic
reference.

With older versions of the agent, you will have to resort to the manual start procedure
for the agent daemon:

/opt/datadog-agent/agent/agent.py -b start

As well as the manual call to stop it:

/opt/datadog-agent/agent/agent.py stop

If you want to run the agent in the foreground, please omit the -b switch.

If you wish to override any configuration setting defined in the config file, you
may resort to environment variables as follows:

DD_LOG_LEVEL=debug ./agent.py start

Running dogstatsd

Dogstatsd allows collecting and submitting custom metrics to datadog. It listens on
a UDP port and statsd metrics may be submitted to it. These will then be relayed
to Datadog.

Dogstatsd relies on the same configuration file defined for the agent where a dogstatsd
configuration section is available. The dogstatsd server will typically run within the
same agent process, but should you need a dedicated process it may also be launched in
standalone mode.

To enable dogstatsd, simply edit /etc/datadog-agent/datadog.yaml and set the relevant
configuration options.

dogstatsd:                        # Dogstatsd configuration options
  enabled: true                   # disabled by default
  bind_host: localhost            # address we'll be binding to
  port: 8125                      # dogstatsd UDP listening port
  non_local_traffic: false        # listen to non-local traffic

Integrations

System Integrations

The following system-level integrations are enabled by default:

• CPU
• Filesystem
• IOStat
• Load
• Memory
• Uptime

Bundled Integrations

Additional integrations currently available:

• Disk
• LPARstats
• Network
• Process

For bundled integrations, a configuration file should be put in place to enable
the integration. Some of these, like the network check, might already be enabled
by default. These configuration files should be found in ./etc/datadog-agent/conf.d.
The name of the YAML configuration file should match that of the integration:
./etc/datadog-agent/conf.d/foo.yaml will enable integration foo, and set its
configuration.

If changes are made to an agent integration, the agent will have to be restarted,
configuration changes are not picked up automatically.

These integrations are shipped as python wheels. You may develop your own should
yout need to, all you have to do is follow the blueprint set by the bundled wheels
here.

See more in the developer notes here.

Uninstall

To remove an installed agent you will run a similar installp command:

installp -e dd-aix-uninstall.log -uv datadog-unix-agent

Note how we're again logging to dd-aix-install.log, you may skip that by removing the -e switch.

Removing Older Agents

Deprecated

The following instructions should only apply to early adopters who may have installed
early dev versions of the unix agent. The following does NOT apply to agent installs
that used the BFF package. Kept here for historic reference.

If you had used the previous scripted installer to install a previous early-development version of the
agent, the former location was /opt/datadog/datadog-unix-agent, you will have to remove that manually.
Please be mindful to preserve you configurations from that setup if you wish to reuse them with the new
agent. You can use the same files, and drop them into the same relative paths in /opt/datadog-agent.

The reason the location was modified was to provide a consistent location across agent versions and
platforms, to match the user experience in Agent 5 and Agent 6.

This should allow you to disable the old running agent and safely delete the old location:

cd /opt/datadog/datadog-unix-agent
. ./venv/bin/activate
./agent.py stop
deactivate
cd ~
# remember to backup config files if required
rm -rf /opt/datadog/datadog-unix-agent

RPM dependencies installed by the scripted installer are no longer required and may be removed if you
so wish. This is the list of former RPM requirements:

• ca-certificates-2016.10.7-2.aix6.1.ppc.rpm
• curl-7.52.1-1.aix6.1.ppc.rpm
• db-4.8.24-3.aix6.1.ppc.rpm
• gdbm-1.8.3-5.aix5.2.ppc.rpm
• gettext-0.19.7-1.aix6.1.ppc.rpm
• glib2-2.14.6-2.aix5.2.ppc.rpm
• python-2.7.10-1.aix6.1.ppc.rpm
• python-devel-2.7.10-1.aix6.1.ppc.rpm
• python-iniparse-0.4-1.aix6.1.noarch.rpm
• python-pycurl-7.19.3-1.aix6.1.ppc.rpm
• python-tools-2.7.10-1.aix6.1.ppc.rpm
• python-urlgrabber-3.10.1-1.aix6.1.noarch.rpm
• readline-6.1-2.aix6.1.ppc.rpm

Developer Notes

The agent runs on Python3. You will typically want setuptools, wheel and virtualenv
on your python development environment. We also have some development tools requirements,
you may install them with:

pip install -r requirements-dev.txt

You may mostly work on this repo from any *nix environment. We do have some python
binary wheel dependencies, on linux these are typically provided as manylinux
pre-built wheels so just installing the requirements should work. Other platforms
(like macOS) might also have pre-built wheels. And in the more popular environments,
even if compiled wheels are not available, building the wheels during the pip
installation should be seamless.

pip install -r requirements.txt

Building

Omnibus build

This is the recommended way for building the agent. Hopefully setting up omnibus on
the builder will be scripted, but until then getting the builder ready is a manual
process.

Platforms

You will need a build machine that matches the target platform, thus:

• AIX 6.1
• AIX 7.1
• AIX 7.2

Omnibus Requirements

To setup omnibus on the target machine you will need:

• AIX Linux toolkit
• gcc (>= 6.3.0 via yum)
• coreutils (via yum - provided with the linux toolkit)
• sudo (via yum)
• libffi and libffi-devel (via yum): required to bootstrap ruby.
• ruby and ruby-devel (via yum)
• GNU tar (via yum)
• bundler (via gem install bundler)

Now let's install the omnibus dependencies, navigate to omnibus/:

• bundle install: when the bundle get to libyajl it will fail on AIX, don't worry we
  have a workaround. Please read on.

On AIX You will need to install a modified version of libyajl before proceeding (you can
run these commands in a scratch directory somewhere):

• libyajl2-gem @ branch jaime/aix
  • yajl:
    you will have to check yajl out in this location libyajl2-gem/ext/libyajl/vendor/yajl
• bundle install
• rake prep
• rake gem
• gem install --local ./pkg/libyajl-1.2.0.gem

Once these steps are complete, you may go back to omnibus/ in the agent repo and run
bundle install or bundle update once again.

If you got here you're doing good and you're almost ready to go.

Build Runtime Requirements + Troubleshooting

• Make sure the ulimits are high enough if you receive out of memory errors (check them with ulimit -a):
  • ulimit -m unlimited (for max memory size)
  • ulimit -s unlimited (for stack size)
  • ulimit -d unlimited (for data seg size)
• Make sure you have enough space in /opt for the requirements + build. (check with df -m, increase with chfs -a size=+2G /opt)
• Make sure you have enough space in /var for the omnibus build
• Omnibus uses git, make sure you have configured git username, etc.
• If you have issues installing gcc 6.3.0, you might need to manually remove gcc-locale
  before attempting the upgrade if gcc is already installed on your system..
• Set the PATH: export PATH="/opt/freeware/bin:$PATH"
• Set the CONFIG_SHELL: export CONFIG_SHELL="/bin/bash"
• Set the TERM: for convenience export TERM=xterm

Note: please note that the omnibus-ruby installer currently requires the datadog-5.5.0-aix
branch on AIX. That will probably be merged to master, but currently is required. No need
for the developer to do anything as that specific branch is the current default for AIX builds.

Note: You can override omnibus-ruby and omnibus-software gem versions if necessary with
the OMNIBUS_RUBY_VERSION and OMNIBUS_SOFTWARE_VERSION respectively - followed by a
bundle update.

Building

Triggering a build is the easiest part, we just need to specify a few more env vars currently:

• JMXFETCH_VERSION: JMXFetch version to bundle with the agent (otherwise the default is used).
• JMXFETCH_HASH: SHA256 hash for the JMXFetch artifact (idem).
• PYTHON_VERSION: 2 or 3 - defaults to 3. Version 2 will be deprecated.

Typically:

JMXFETCH_VERSION="<version>" JMXFETCH_HASH="<hash>" PYTHON_VERSION="3" bundle exec omnibus build agent --log-level=info

Note: JMXFetch is included in the build but is not officially supported (the AIX Agent does not
include the facilities to configure and start JMXFetch).

Deprecated: Scripted Installer

This build method has been deprecated in favor of the omnibus build. Keeping here
for historical reasons.

We provide a script in packaging named ./packaging/builder that will allow you
to build a self-extractable installer. You may use it as follows:

./builder -b {github_branch} -v {version}

Where github_branch would be a remote branch in your repository and version
the version for the agent release. This will result in a ksx self-extractable
installer that should just work across supported AIX environments.

Integrations

The agent has two types of checks or integrations.

• Core checks: built into the agent.
• Wheel checks: additional integrations we may package, bundle and install on an
  agent environment.

Here, we will mostly discuss wheel checks, as they provide the natural facilities to
extend the agent.

All checks (both core and whell) depend on the AgentCheck base class, shipped in the
project's checks.agent_check module, and will inherit from it. This base class will
ship with the agent and can be considered part of the environment, wheel checks can
expect its availability.

The pattern for the wheel checks has been taken from integrations-core,
so if you're familiar with that, you should be able to hit the ground running.

A check wheel foo should have the following skeleton:

checks/bundled/foo/README.md                                    # check README.
checks/bundled/foo/setup.py                                     # python package setup.
checks/bundled/foo/tests/                                       # tests - we use pytest.
checks/bundled/foo/datadog_checks/__init__.py                   # namespace module init - note: this file is not shipped with the wheel. 
checks/bundled/foo/datadog_checks/foo/__about__.py              # foo module about - includes version info.
checks/bundled/foo/datadog_checks/foo/__init__.py               # foo module init.
checks/bundled/foo/datadog_checks/foo/foo.py                    # foo check implementation.
checks/bundled/foo/datadog_checks/foo/data/conf.yaml.example    # sample configuration.
checks/bundled/foo/pytest.ini                                   # any additional pytest config.
checks/bundled/foo/requirements-dev.txt                         # (optional) any additional dev requirements.

Please take a look at any of the bundled checks here
for inspiration.

AIX

If you wish to develop directly on an AIX rig, we recommend the following development
requirements be met. We will assume these conditions are met when we discuss AIX
development workflows:

• yum (use script here: https://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/yum.sh)
• python3.6+ (should be installed with the yum script)
• python-tools (should be installed with the yum script)
• gcc (install with yum)
• virtualenv (recommended)

Should you need to build a binary wheel, you would typically build and compile it as follows:

CC="gcc -lgcc"  LDSHARED="/opt/freeware/lib/python3.6/config/ld_so_aix gcc -bI:/opt/freeware/lib/python2.7/config/python.exp" CFLAGS="-fno-strict-aliasing -Wall  -Wstrict-prototypes -fPIC -O2" python setup.py bdist_wheel

The command above is a blueprint, but might require some tweaking as far as the include (-I) and
lib (-L) directives may go.

If the command succeeds you'll typically find the compiled wheel in the python-package dist/
directory.

Please let us know if you have any questions or issues!

Happy DataDoggin'!