systemd Deep Dive: Linux Service Management

Introduction

systemd is the system and service manager for Linux operating systems. It has become the default initialization system for most modern Linux distributions including RHEL, CentOS, Fedora, Ubuntu (15.04+), and Debian. Understanding systemd is essential for any Linux system administrator or developer working with Linux servers.

Before systemd, SysV init was the standard, using numbered runlevels and sequential startup scripts. systemd was created to address the limitations of SysV init, offering parallel startup, dependency management, socket activation, and centralized logging.

This comprehensive guide explores systemd in depth: its architecture, unit types, service management, logging with journald, and troubleshooting techniques.

Understanding systemd Architecture

The systemd Philosophy

systemd approaches system initialization with several key principles:

Parallelization: Services start concurrently, not sequentially
Socket Activation: Services start on-demand when their socket receives a connection
Dependency Management: Automatic handling of service dependencies
Resource Control: Cgroups for process isolation and resource limits
Unified Logging: Centralized logging through journald

The Unit System

The core concept in systemd is the unit - a configuration file that describes a resource or action. Units are defined in unit files.

Unit Types

systemd recognizes several unit types:

Unit Type	Description	File Extension
Service	Daemon/process	.service
Socket	IPC socket	.socket
Target	Group of units	.target
Device	Hardware device	.device
Mount	Filesystem mount	.mount
Automount	Auto-mount point	.automount
Path	Path-based activation	.path
Timer	Scheduled activation	.timer
Snapshot	Saved unit state	.snapshot
Slice	Process group resources	.slice
Scope	External process group	.scope

Service Units Deep Dive

Service units define how systemd manages daemon processes.

Basic Service Unit File

[Unit]
Description=My Application Service
Documentation=https://example.com/docs
After=network.target
Wants=network.target

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/opt/app
ExecStart=/opt/app/server --config /etc/app/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStartSec=30
TimeoutStopSec=60

# Resource limits
MemoryMax=512M
CPUQuota=50%
LimitNOFILE=1024

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

Service Types

Different service types handle daemon processes differently:

# Type=simple (default)
# Main process doesn't fork
# systemd starts and manages the process directly

[Service]
Type=simple
ExecStart=/usr/bin/myapp

# Type=forking
# Main process forks child processes
# systemd waits for original to exit

[Service]
Type=forking
ExecStart=/usr/bin/myapp --daemon

# Type=oneshot
# Runs once and exits
# Useful for one-time tasks

[Service]
Type=oneshot
ExecStart=/usr/bin/setup-script
RemainAfterExit=yes

# Type=notify
# Similar to simple but sends notification when ready
# Uses sd_notify() in the application

[Service]
Type=notify
ExecStart=/usr/bin/myapp

Dependency Management

[Unit]
# Start after network is ready
After=network-online.target network.target

# Start before shutdown
Before=shutdown.target reboot.target

# Weak dependency - start if needed, but don't fail if not available
Wants=network-online.target

# Strong dependency - must be running
Requires=network-online.target

Restart Policies

[Service]
# Restart options
Restart=no              # Don't restart
Restart=on-success       # Restart on clean exit
Restart=on-failure     # Restart on abnormal exit (default)
Restart=on-abnormal     # Restart on signal, timeout, watchdog
Restart=always          # Always restart

# Restart delay
RestartSec=5            # Wait 5 seconds before restarting

# Start limits
StartLimitInterval=1min  # Max restarts in interval
StartLimitBurst=5       # Allow 5 restarts in interval

Resource Limits

[Service]
# Memory limits
MemoryMax=1G
MemoryHigh=800M

# CPU limits
CPUQuota=50%

# Process limits
LimitNOFILE=65536
LimitNPROC=4096

# I/O limits
IOWeight=100
IOReadBandwidthMax=100M
IOWriteBandwidthMax=100M

Target Units

Targets are similar to SysV runlevels - they group units together.

Common Targets

Target	Description	Similar to Runlevel
graphical.target	Graphical interface with multi-user	5
multi-user.target	Multi-user text mode	3
rescue.target	Single-user rescue mode	1
emergency.target	Emergency shell	S
shutdown.target	System shutdown	-
default.target	Default target (usually graphical.target)	-

Creating Custom Target

# /etc/systemd/system/myapp.target

[Unit]
Description=My Application Target
Documentation=https://example.com
After=network-online.target
Wants=network-online.target

[Install]
WantedBy=multi-user.target

Managing Targets

# Set default target
systemctl set-default multi-user.target

# Get current target
systemctl get-default

# Switch to different target (without reboot)
systemctl isolate graphical.target

# List targets
systemctl list-units --type=target

Socket Units

Socket units enable on-demand service activation.

Example Socket Unit

# /etc/systemd/system/myapp.socket

[Unit]
Description=My Application Socket
Documentation=https://example.com
PartOf=myapp.service

[Socket]
ListenStream=/run/myapp.sock
SocketMode=0660
SocketUser=appuser
SocketGroup=appgroup

# Accept=true for multi-connection sockets
Accept=false

[Install]
WantedBy=sockets.target

Associated Service

# /etc/systemd/system/myapp.service

[Unit]
Description=My Application
Requires=myapp.socket
After=myapp.socket

[Service]
Type=notify
ExecStart=/usr/bin/myapp
SocketBindDgram=auto

[Install]
WantedBy=multi-user.target

Timer Units

Timers replace cron for scheduled tasks.

Timer Example

# /etc/systemd/system/backup.timer

[Unit]
Description=Daily Backup Timer
Documentation=https://example.com

[Timer]
# Run daily at 3am
OnCalendar=daily
Persistent=true

# Random delay to avoid thundering herd
RandomizedDelaySec=1h

[Install]
WantedBy=timers.target

Timer with Specific Schedule

[Timer]
# Every Monday at 9am
OnCalendar=Mon *-*-* 09:00:00

# Every 15 minutes
OnCalendar=*:0/15

# Every hour
OnCalendar=hourly

# Specific date
OnCalendar=2026-01-01 00:00:00

# Multiple schedules
OnCalendar=Mon *-*-1..7 09:00:00  # First Monday of month

Service for Timer

# /etc/systemd/system/backup.service

[Unit]
Description=Backup Service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

[Install]
WantedBy=multi-user.target

Using systemctl

Service Management Commands

# Start, stop, restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx

# Check status
systemctl status nginx
systemctl is-active nginx
systemctl is-enabled nginx

# Enable/disable at boot
systemctl enable nginx
systemctl disable nginx

# Mask/unmask (completely disable)
systemctl mask nginx
systemctl unmask nginx

# View all services
systemctl list-units --type=service
systemctl list-unit-files --type=service

Viewing Dependencies

# View service dependencies
systemctl list-dependencies nginx.service

# Reverse dependencies (what depends on this)
systemctl list-dependencies --reverse nginx.service

Viewing and Editing Unit Files

# View unit file
systemctl cat nginx.service

# Edit unit file (creates override)
systemctl edit nginx.service

# Full edit
systemctl edit --full nginx.service

# Reload systemd after changes
systemctl daemon-reload

Journald and Logging

journald is systemd’s logging component.

Using journalctl

# View all logs
journalctl

# View current boot logs
journalctl -b

# View previous boot
journalctl -b -1

# Follow logs in real-time
journalctl -f

# Filter by service
journalctl -u nginx.service

# Filter by time
journalctl --since "1 hour ago"
journalctl --since "2026-01-01" --until "2026-01-02"

# Filter by priority (0=emerg to 7=debug)
journalctl -p err
journalctl -p warning

# Show only errors since last boot
journalctl -b -p err

# Disk usage
journalctl --disk-usage

# Rotate logs
journalctl --vacuum-size=500M
journalctl --vacuum-time=7d

Structured Logging

# View with specific fields
journalctl -o json-pretty

# Filter by field
journalctl _SYSTEMD_UNIT=nginx.service
journalctl _PID=1234

# List unique values for field
journalctl -F _SYSTEMD_UNIT

Adding Journal to Unit

[Service]
# Send to journal with syslog
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

# Or use syslog
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=myapp

System State Management

Power Operations

# Reboot
systemctl reboot

# Power off
systemctl poweroff

# Suspend
systemctl suspend

# Hibernate
systemctl hibernate

# Hybrid sleep
systemctl hybrid-sleep

Systemd Analyze

# Analyze boot time
systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain nginx.service

# Plot boot chart
systemd-analyze plot > boot.svg

# Verify unit files
systemd-analyze verify nginx.service

Troubleshooting

Common Issues and Solutions

Service Failed to Start

# Check detailed status
systemctl status nginx
journalctl -u nginx -n 50

# Check configuration
systemctl cat nginx.service
nginx -t

# View kernel messages
dmesg | tail

Dependency Failures

# Check what failed
systemctl list-dependencies --failed

# Check job status
systemctl list-jobs

# Clean up failed state
systemctl reset-failed

Resource Limits Hit

# Check if hitting limits
systemctl status nginx
journalctl -u nginx | grep -i "resource\|limit\|memory\|cpu"

# Temporarily remove limits
systemctl edit nginx.service
# Remove MemoryMax, CPUQuota lines
systemctl daemon-reload
systemctl restart nginx

Socket Activation Issues

# Check socket status
systemctl status myapp.socket

# Check socket backlog
ss -l | grep myapp

# Check pending connections
systemctl list-sockets

Debugging Service Startup

# Run service in foreground for debugging
systemd-run -t --scope /usr/bin/myapp

# Add debug to service
systemctl edit nginx.service
# Add:
# Environment=SYSTEMD_LOG_LEVEL=debug

# Restart and watch
systemctl restart nginx
journalctl -u nginx -f

Advanced Configuration

Drop-in Overrides

# Create override directory
mkdir -p /etc/systemd/system/nginx.service.d

# Create override file
cat > /etc/systemd/system/nginx.service.d/override.conf <<EOF
[Service]
MemoryMax=1G
Restart=always
EOF

# Reload and restart
systemctl daemon-reload
systemctl restart nginx

User Services

# User service directory
mkdir -p ~/.config/systemd/user

# Create user service
cat > ~/.config/systemd/user/myapp.service <<EOF
[Unit]
Description=My App

[Service]
ExecStart=/home/user/myapp
Restart=on-failure

[Install]
WantedBy=default.target
EOF

# Enable user service
systemctl --user enable myapp.service
systemctl --user start myapp.service

# User persist across services sessions
loginctl enable-linger user

Environment Variables

[Service]
Environment=APP_ENV=production
Environment=APP_PORT=8080
EnvironmentFile=-/etc/app/env.conf  # - means optional

[Service]
ExecStart=/usr/bin/myapp $APP_ARGS

Best Practices

Service Unit Design

[Unit]
# Always include
Description=Meaningful description
Documentation=URL to documentation

# Proper ordering
After=network.target
Wants=network-online.target

[Service]
# Use appropriate type
Type=simple  # Most common

# Always set restart
Restart=on-failure
RestartSec=5

# Set timeouts
TimeoutStartSec=60
TimeoutStopSec=30

# Logging
StandardOutput=journal
StandardError=journal

# Security
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/log/myapp

[Install]
WantedBy=multi-user.target

Security Hardening

[Service]
# Restrict capabilities
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

# Filesystem restrictions
ProtectSystem=strict
ProtectHome=true
ProtectTmp=true
PrivateTmp=true
PrivateDevices=true
PrivateUsers=true

# System call filtering
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

# Resource limits
MemoryDenyWriteExecute=true

Conclusion

systemd is the foundation of modern Linux system management. Understanding its architecture - units, services, targets, socket activation, timers, and journald - is essential for Linux administrators. This guide covered the core concepts, practical configuration, troubleshooting, and best practices for working with systemd.

Master systemd and you’ll be able to manage services effectively, troubleshoot issues quickly, and leverage powerful features like on-demand activation and centralized logging.