strace: Linux System Call Tracer Complete Guide

strace is a Linux command that traces system calls and signals made by a process. It can track a process’s execution based on PID, show which system calls it makes, and perform statistical analysis on their execution. It’s commonly used to troubleshoot program faults, debug issues, or identify reasons for slow performance.

Introduction

Every interaction between a program and the Linux kernel happens through system calls. When a program reads a file, writes to network, allocates memory, or creates a new process, it makes a system call. strace captures these calls, providing invaluable insight into what a program is actually doing at the kernel level.

strace attaches to running processes or launches new ones, intercepting and recording system calls and signals. This makes it indispensable for debugging mysterious crashes, understanding program behavior, identifying performance bottlenecks, and even learning how operating systems work.

The tool works at the user-kernel boundary, capturing the arguments and return values of each system call. This information reveals program behavior that source code inspection alone might not show, including interactions with the filesystem, network, and system resources.

Basic Usage

Tracing a Command

The simplest use of strace runs a command and shows all system calls:

strace ls /tmp

This runs ls /tmp and displays every system call made during execution. The output shows the system call name, its arguments in parentheses, and its return value.

Sample output:

execve("/bin/ls", ["ls", "/tmp"], 0x7ffd...) = 0
brk(NULL)                               = 0x55f8c000
access("/etc/ld.so.preload", R_OK)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=78649, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0...") = 832
...

Tracing with Timestamps

Add timestamps to understand when calls occur:

# Absolute timestamps
strace -t ls /tmp

# Timestamps with microseconds
strace -tt ls /tmp

# Relative time from previous call
strace -r ls /tmp

Output with relative timing:

0.000000 execve("/bin/ls", ["ls", "/tmp"], 0x7ffd...) = 0
0.000153 brk(NULL)                               = 0x55f8c000
0.000098 access("/etc/ld.so.preload", R_OK)     = -1 ENOENT
0.000102 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3

Output to File

strace output can be verbose. Redirect to a file for analysis:

strace -o /tmp/strace_output.txt ls /tmp

Later analyze the file with grep, awk, or other tools.

Advanced Usage

Attaching to Running Processes

Trace an existing process by PID:

strace -p 16742

Press Ctrl+C to stop tracing.

Combine with options:

strace -o /tmp/output -f -r -s 4096 -p 16742

Flags explained:

• -o /tmp/output: Write output to file
• -f: Follow child processes (trace forks)
• -r: Print relative timestamps between calls
• -s 4096: Limit string output to 4096 characters
• -p 16742: Attach to process with PID 16742

Following Child Processes

When programs fork child processes, use -f to trace them:

strace -f -o /tmp/strace.txt ./my_program

The -ff option creates separate files for each PID, useful for complex forking.

Filtering System Calls

Focus on specific call types:

# Only file operations
strace -e trace=file ls /tmp

# Only network operations
strace -e trace=network curl example.com

# Only memory operations
strace -e trace=memory ls

# Multiple categories
strace -e trace=file,network,process curl example.com

# Specific calls
strace -e trace=openat,read,write ls /tmp

Statistical Summary

Get an overview of which system calls are most used:

strace -c ls /tmp

Output:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0         1           execve
  0.00    0.000000           0        12           brk
  0.00    0.000000           0         5         access
  0.00    0.000000           0         6        openat
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000            76        69           total

The -C option shows both the trace and the summary.

Common Options Reference

Practical Use Cases

Debugging Program Crashes

When a program crashes without error messages, strace often reveals the last system call:

strace ./crashing_program 2>&1 | tail -20

Look for error return values (negative numbers like -1) and error messages (ENOENT, EACCES, etc.) near the end.

Identifying Permission Issues

Debug “Permission denied” errors:

strace -e openat,access ./program 2>&1 | grep -i denied

This shows exactly which files the program tries to access.

Analyzing Performance

Find slow system calls:

strace -T -c ./slow_program

The time column shows which calls take longest.

Monitoring Network Activity

See network connections:

strace -e trace=network curl https://example.com

Understanding Program Behavior

Learn what a program does:

strace -e trace=openat,read,write cat /etc/passwd

This shows file operations without the program overhead.

Important Notes

Permissions

strace requires appropriate permissions:

• Trace your own processes: Normal user access
• Trace system processes: Requires sudo
• Trace setuid programs: Often restricted

sudo strace -p 1  # Attach to init/systemd

Performance Impact

Tracing significantly slows execution:

• Attach to production processes carefully
• Consider using -f only when needed
• Shorter trace durations reduce impact

Library Calls

For library calls (not system calls), use ltrace:

ltrace ./program

ltrace intercepts calls to shared libraries.

Filtering Noise

System calls from shared libraries can create noise. Filter with -e:

# Only trace your program's calls, not libraries
strace -e trace=file,process,network ./program

Examples from Real Debugging

Finding Configuration Files

strace -e openat ./program 2>&1 | grep config

Shows all config files the program opens.

Debugging Connection Issues

strace -e network,connect curl https://api.example.com

See DNS lookups, connections, and data transfer.

Analyzing File Access Patterns

strace -e openat,read,write,close -c ./program

Summary shows files used most frequently.

Investigating Startup Failures

strace -f -o startup.strace ./program 2>&1

Search startup.strace for “ENOENT” (file not found) errors.

Conclusion

strace is an indispensable tool for Linux system debugging and analysis. It provides visibility into exactly what programs do at the kernel level, revealing issues that might be invisible through other means. Whether you’re debugging a crash, investigating performance problems, understanding program behavior, or learning about Linux internals, strace delivers insights unavailable through any other method.

The key to effective strace use is filtering—there’s usually far more information than needed. Learn to combine options like -e trace=, -c, and output redirection to focus on exactly what you need to solve your specific problem.

References

• strace Man Page
• Brendan Gregg: strace
• IBM DeveloperWorks: System Calls
• Linux.cn: strace Tutorial