[Bug]: OTEL plugin fails to start - "Failed to create logs service" journal creation error #21037
Description
Bug description
Description
The OTEL plugin fails to initialize and cannot create journal logs in any directory, preventing OpenTelemetry metric collection from working entirely.
Actual Behavior
The plugin fails with journal creation error:
/usr/libexec/netdata/plugins.d/otel-plugin --otel-logs-journal-dir /var/log/netdata/otel/v1 --otel-endpoint 0.0.0.0:4317
Error: Failed to create logs service
Caused by:
0: Failed to create journal log for directory: /var/log/netdata/otel/v1
1: i/o error during object operation: No such file or directory (os error 2)
2: No such file or directory (os error 2)
Additional Details
- Error occurs with any directory path (tested with
/tmp,/var/log/netdata/otel/v1, etc.) - Directories exist and have proper permissions (755, owned by netdata:netdata)
- Plenty of disk space available (3.1T free)
- Plugin binary exists and has correct permissions (
-rwsr-xr-x 1 root root) - All library dependencies are satisfied (verified with
ldd) - Error occurs even when running as root
- The
--otel-logs-journal-dirparameter is required, cannot be omitted
Impact
This completely blocks OpenTelemetry integration. The stable releases don't include the OTEL plugin, so nightly builds are required for OTEL functionality, but the plugin is currently non-functional.
Use Case
Collecting OpenTelemetry metrics from applications and forwarding them to Netdata for monitoring and visualization.
Expected behavior
Expected Behavior
The OTEL plugin should start successfully and listen for OpenTelemetry metrics on port 4317.
Steps to reproduce
Steps to Reproduce
- Install Netdata nightly build v2.6.0-313-nightly
- Configure OTEL plugin in netdata.conf:
[plugins] otel = yes [plugin:otel] update every = 1s command options = --otel-logs-journal-dir /var/log/netdata/otel/v1 --otel-endpoint 0.0.0.0:4317
- Try to run the plugin manually:
/usr/libexec/netdata/plugins.d/otel-plugin --otel-logs-journal-dir /var/log/netdata/otel/v1 --otel-endpoint 0.0.0.0:4317
Installation method
kickstart.sh
System info
## Environment
- **OS:** Ubuntu 22.04 LTS
- **Netdata version:** v2.6.0-313-nightly
- **Installation:** Docker container
- **Architecture:** x86_64
Linux xxxxxxxx 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=22.04
/etc/lsb-release:DISTRIB_CODENAME=jammy
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
/etc/os-release:PRETTY_NAME="Ubuntu 22.04.5 LTS"
/etc/os-release:NAME="Ubuntu"
/etc/os-release:VERSION_ID="22.04"
/etc/os-release:VERSION="22.04.5 LTS (Jammy Jellyfish)"
/etc/os-release:VERSION_CODENAME=jammy
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
/etc/os-release:UBUNTU_CODENAME=jammyNetdata build info
time=2025-09-22T20:47:24.075Z comm=netdata source=daemon level=notice errno="2, No such file or directory" tid=2523492 msg="CONFIG: cannot load user config '/etc/netdata/stream.conf'. Will try stock config."
Packaging:
Netdata Version ____________________________________________ : v2.6.0-313-nightly
Installation Type __________________________________________ : oci
Package Architecture _______________________________________ : x86_64
Package Distro _____________________________________________ : unknown
Configure Options __________________________________________ : cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_STANDARD=11 -DCMAKE_CXX_STANDARD=14 -DBUILD_SHARED_LIBS= -DCMAKE_C_FLAGS='-ffunction-sections -fdata-sections -O2 -funroll-loops -pipe -fexceptions -fstack-protector-strong -D_FORTIFY_SOURCE=3 -fstack-clash-protection -fcf-protection=full -Wno-builtin-macro-redefined -fexceptions -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_CXX_FLAGS=' -ffunction-sections -fdata-sections -O2 -funroll-loops -pipe -fexceptions -fstack-protector-strong -D_FORTIFY_SOURCE=3 -fstack-clash-protection -fcf-protection=full -Wno-builtin-macro-redefined -fexceptions -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_COMPILE_DEFINITIONS='_GNU_SOURCE' -DCMAKE_EXE_LINKER_FLAGS='-Wl,--gc-sections -fexceptions -fstack-protector-strong -D_FORTIFY_SOURCE=3 -fstack-clash-protection -fcf-protection=full -Wno-builtin-macro-redefined -fexceptions -rdynamic' -DCMAKE_SHARED_LINKER_FLAGS='-Wl,--gc-sections'
Default Directories:
User Configurations ________________________________________ : /etc/netdata
Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
Permanent Databases ________________________________________ : /var/lib/netdata
Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
Static Web Files ___________________________________________ : /usr/share/netdata/web
Log Files __________________________________________________ : /var/log/netdata
Lock Files _________________________________________________ : /var/lib/netdata/lock
Home _______________________________________________________ : /var/lib/netdata
Operating System:
Kernel _____________________________________________________ : Linux
Kernel Version _____________________________________________ : 5.15.0-153-generic
Operating System ___________________________________________ : Ubuntu
Operating System ID ________________________________________ : ubuntu
Operating System ID Like ___________________________________ : debian
Operating System Version ___________________________________ : 22.04.5 LTS (Jammy Jellyfish)
Operating System Version ID ________________________________ : 13
Detection __________________________________________________ : /host/etc/os-release
Hardware:
CPU Cores __________________________________________________ : 12
CPU Frequency ______________________________________________ : 3400000000
RAM Bytes __________________________________________________ : 67180756992
Disk Capacity ______________________________________________ : 8001574060032
CPU Architecture ___________________________________________ : x86_64
Virtualization Technology __________________________________ : none
Virtualization Detection ___________________________________ : none
Container:
Container __________________________________________________ : docker
Container Detection ________________________________________ : dockerenv
Container Orchestrator _____________________________________ : none
Container Operating System _________________________________ : Debian GNU/Linux
Container Operating System ID ______________________________ : debian
Container Operating System ID Like _________________________ : unknown
Container Operating System Version _________________________ : 13 (trixie)
Container Operating System Version ID ______________________ : 13
Container Operating System Detection _______________________ : /etc/os-release
Features:
Built For __________________________________________________ : Linux
Netdata Cloud ______________________________________________ : YES
Health (trigger alerts and send notifications) _____________ : YES
Streaming (stream metrics to parent Netdata servers) _______ : YES
Back-filling (of higher database tiers) ____________________ : YES
Replication (fill the gaps of parent Netdata servers) ______ : YES
Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip brotli)
Contexts (index all active and archived metrics) ___________ : YES
Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
Machine Learning ___________________________________________ : YES
Memory Allocator ___________________________________________ : system
Database Engines:
dbengine (compression) _____________________________________ : YES (zstd lz4)
alloc ______________________________________________________ : YES
ram ________________________________________________________ : YES
none _______________________________________________________ : YES
Connectivity Capabilities:
ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
static (Netdata internal web server) _______________________ : YES
WebRTC (experimental) ______________________________________ : NO
Native HTTPS (TLS Support) _________________________________ : YES
TLS Host Verification ______________________________________ : YES
Libraries:
LZ4 (extremely fast lossless compression algorithm) ________ : YES
ZSTD (fast, lossless compression algorithm) ________________ : YES
zlib (lossless data-compression library) ___________________ : YES
Brotli (generic-purpose lossless compression algorithm) ____ : YES
protobuf (platform-neutral data serialization protocol) ____ : YES (system)
OpenSSL (cryptography) _____________________________________ : YES
libdatachannel (stand-alone WebRTC data channels) __________ : NO
JSON-C (lightweight JSON manipulation) _____________________ : YES
libcap (Linux capabilities system operations) ______________ : YES
libcrypto (cryptographic functions) ________________________ : YES
libyaml (library for parsing and emitting YAML) ____________ : YES
libmnl (library for working with netfilter) ________________ : YES
stacktraces (library for getting stack traces) _____________ : libbacktrace (mmap, threads, data)
Plugins:
apps (monitor processes) ___________________________________ : YES
cgroups (monitor containers and VMs) _______________________ : YES
cgroup-network (associate interfaces to CGROUPS) ___________ : YES
proc (monitor Linux systems) _______________________________ : YES
tc (monitor Linux network QoS) _____________________________ : YES
diskspace (monitor Linux mount points) _____________________ : YES
freebsd (monitor FreeBSD systems) __________________________ : NO
macos (monitor MacOS systems) ______________________________ : NO
windows (monitor Windows systems) __________________________ : NO
statsd (collect custom application metrics) ________________ : YES
timex (check system clock synchronization) _________________ : YES
idlejitter (check system latency and jitter) _______________ : YES
bash (support shell data collection jobs - charts.d) _______ : YES
debugfs (kernel debugging metrics) _________________________ : YES
cups (monitor printers and print jobs) _____________________ : NO
ebpf (monitor system calls) ________________________________ : NO
freeipmi (monitor enterprise server H/W) ___________________ : YES
network-viewer (monitor TCP/UDP IPv4/6 sockets) ____________ : YES
systemd-journal (monitor journal logs) _____________________ : YES
windows-events (monitor Windows events) ____________________ : NO
nfacct (gather netfilter accounting) _______________________ : NO
perf (collect kernel performance events) ___________________ : YES
slabinfo (monitor kernel object caching) ___________________ : YES
Xen ________________________________________________________ : NO
Xen VBD Error Tracking _____________________________________ : NO
Exporters:
AWS Kinesis ________________________________________________ : NO
GCP PubSub _________________________________________________ : NO
MongoDB ____________________________________________________ : YES
Prometheus (OpenMetrics) Exporter __________________________ : YES
Prometheus Remote Write ____________________________________ : YES
Graphite ___________________________________________________ : YES
Graphite HTTP / HTTPS ______________________________________ : YES
JSON _______________________________________________________ : YES
JSON HTTP / HTTPS __________________________________________ : YES
OpenTSDB ___________________________________________________ : YES
OpenTSDB HTTP / HTTPS ______________________________________ : YES
All Metrics API ____________________________________________ : YES
Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
Trace All Netdata Allocations (with charts) ________________ : NO
Developer Mode (more runtime checks, slower) _______________ : NO
Runtime Information:
Profile ____________________________________________________ : standalone
Stream Parent (accept data from Children) __________________ : NO
Stream Child (send data to a Parent) _______________________ : NO
Total System Memory ________________________________________ : 4294967296
Available System Memory ____________________________________ : 4111134720Additional info
No response