Solaris Performance and Monitoring tools

Here’s a detailed overview of key performance and monitoring tools in Solaris, categorized by functionality:

1. General System Monitoring Tools

Tool	Description	Common Usage
`prstat`	Similar to Linux `top`. Displays active processes and CPU/memory usage dynamically.	`prstat -a` → Show per-user and per-process statistics.
`mpstat`	Reports CPU usage per processor. Helps detect CPU bottlenecks.	`mpstat 5` → Refresh every 5 seconds.
`vmstat`	Reports system-wide virtual memory, process, CPU, and paging statistics.	`vmstat 5` → Monitor performance trends.
`iostat`	Displays I/O statistics for devices and partitions. Useful for disk performance tuning.	`iostat -xn 5` → Extended output every 5 seconds.
`sar`	Collects, reports, and saves system activity data. Can report on CPU, I/O, memory, etc.	`sar -u 5 5` → CPU utilization 5 times every 5 sec.
`uptime`	Quick summary of system load averages and uptime.	`uptime` → Shows 1, 5, 15-minute load averages.
`dtrace`	Dynamic tracing framework for kernel and user-space debugging/performance tuning.	Used for detailed custom tracing scripts.

2. CPU Performance Tools

Tool	Description
`mpstat`	Monitors per-CPU utilization and identifies load imbalance across CPUs.
`psrinfo`	Displays processor status, speed, and online/offline information.
`psradm`	Used to enable/disable CPUs. Useful for performance testing or power management.
`cpustat`	Collects detailed CPU performance counter data (cache misses, cycles, instructions).
`pbind`	Binds a process to a specific CPU (for performance tuning).

3. Memory and Paging Tools

Tool	Description
`vmstat`	Monitors virtual memory activity, paging, and process queues.
`sar -r`	Reports memory and swap space utilization.
`swap`	Shows current swap space usage and configuration.
`mdb`	Modular debugger, can inspect kernel memory data structures (advanced).

4. Disk and File System Tools

Tool	Description
`iostat`	Monitors disk I/O performance. Extended options show service times and queue lengths.
`df` / `du`	Reports file system usage.
`fsstat`	Reports file system-level statistics (reads, writes, operations).
`metastat`	For monitoring Solaris Volume Manager disk sets and RAID volumes.
`zpool` / `zfs`	For monitoring and managing ZFS file systems and pools.

5. Network Performance Tools

Tool	Description
`netstat`	Displays network connections, routing tables, interface stats, and protocol usage.
`snoop`	Solaris-native packet sniffer (like `tcpdump` on Linux).
`dlstat`	Monitors network data-link activity (bandwidth, throughput).
`kstat`	Provides kernel statistics — can display NIC statistics and error counters.
`ndd`	Views and sets network driver and protocol parameters.
`ping`, `traceroute`	Basic connectivity and latency measurement tools.

6. Comprehensive and Advanced Monitoring Tools

Tool	Description
`DTrace`	The most powerful Solaris performance framework. Enables dynamic tracing of kernel and user-space events. Used for deep system diagnosis.
`kstat`	Kernel statistics access command; allows fine-grained performance metrics.
`PerfMon` / `Sun Management Center`	GUI-based system performance monitoring and alerting tools (legacy but still used in enterprise environments).
`svcs` / `svcprop`	For monitoring SMF (Service Management Facility) services — system service status and dependencies.
`fmadm` / `fmstat`	For monitoring the Fault Management Architecture (FMA) — detects and diagnoses hardware faults.

7. Logging and Auditing

Tool	Description
`/var/adm/messages`	General system log for kernel and hardware messages.
`dmesg`	Displays recent kernel messages (boot and hardware-related).
`logadm`	Manages and rotates system logs.
`auditreduce` / `praudit`	Used for viewing and filtering audit trail logs.

8. Example Performance Diagnosis Workflow

A typical Solaris performance analysis session might involve:

prstat -a                     # Check CPU usage per process/user
mpstat 5                      # See per-CPU utilization
vmstat 5                      # Monitor memory, swap, and run queue
iostat -xn 5                  # Monitor disk throughput and latency
netstat -i                    # Check network interface stats
zpool iostat 5                # If using ZFS, check storage pool I/O
sar -u 5 5                    # Log CPU usage samples

For deeper analysis:

dtrace -n 'syscall:::entry { @[probefunc] = count(); }'  # Trace syscalls

Would you like me to include example DTrace scripts (for CPU, memory, or I/O tracing)?
They’re often the most powerful part of Solaris performance tuning.

Solaris a technically exceptional UNIX.

Many of Solaris’s performance and monitoring tools later inspired Linux utilities, but some remain unique to Solaris (either entirely, or in how deeply they integrate with the kernel).

Here’s a breakdown:

Tools and Features Unique (or Originally Native) to Solaris

1. DTrace (Dynamic Tracing Framework)

Unique to Solaris.
Description:
DTrace is a kernel-level and user-level dynamic tracing framework for live system instrumentation. It can safely trace any event in real time — system calls, function calls, I/O, scheduler behavior, etc. — without rebooting or recompiling.
Why it’s special:
It allows deep observability with almost zero overhead when inactive.
Linux equivalent:
Later attempts to emulate it include:
- SystemTap (inspired by DTrace, less stable and flexible)
- eBPF / bpftrace (modern and close in spirit but architecturally different)
Conclusion:
DTrace was a Solaris original, and Linux has only recently approached its level with eBPF (15+ years later).

2. FMA (Fault Management Architecture)

Unique to Solaris.
Tools: fmadm, fmstat
Description:
A kernel-level predictive self-healing framework that detects, diagnoses, and isolates hardware faults (CPU, memory, I/O, etc.) before they cause system crashes.
Why it’s special:
FMA was part of the “Predictive Self-Healing” architecture in Solaris 10. It could automatically offline faulty components (like CPUs or DIMMs) dynamically.
Linux equivalent:
None fully equivalent. Linux has some hardware error handling via mcelog or RAS Daemon, but not a unified predictive framework like FMA.
Conclusion:
Completely Solaris-unique in scope and integration.

3. Service Management Facility (SMF)

Tools: svcs, svcadm, svcprop
Description:
Framework for managing system services with dependencies, restart policies, and states.
Why it’s special:
Replaced traditional init.d scripts with dependency-aware, fault-tolerant service management.
Linux equivalent:
systemd (modern Linux init system) was heavily inspired by SMF concepts, though developed independently.
Conclusion:
SMF is a Solaris innovation, later mirrored in Linux via systemd.

4. ZFS and Related Tools

Tools: zpool, zfs, fsstat
Description:
Advanced 128-bit filesystem with integrated volume management, snapshots, and checksumming.
Why it’s special:
ZFS was revolutionary: it unified file system + volume manager + data integrity.
Linux equivalent:
OpenZFS on Linux exists now, but it was originally developed for Solaris.
It was copied/forked from the OpenSolaris codebase.
Conclusion:
ZFS is a Solaris-born technology that Linux later adopted externally.

5. kstat / kstat Interface

Description:
Solaris’s kernel statistics interface for retrieving detailed kernel metrics (per-device, per-driver, etc.).
Tools: kstat, kstat(3KSTAT) API.
Why it’s special:
Provides consistent, hierarchical kernel metrics access — used by many Solaris monitoring tools internally.
Linux equivalent:
Linux has /proc and /sys but no unified kernel statistics API equivalent to kstat.
Conclusion:
Unique to Solaris, though conceptually related to /proc.

6. kvmstat, cpustat, pbind, psrinfo

Description:
Tools that give low-level access to CPU hardware counters, processor binding, and kernel VM stats.
Why it’s special:
Very tight integration with Solaris kernel subsystems.
Linux equivalent:
Some analogues exist (numactl, taskset, perf), but Solaris’s CPU observability was deeper and more unified.
Conclusion:
Originally Solaris-only, partially imitated in Linux perf tools.

7. snoop (Solaris-native packet sniffer)

Description:
Built-in packet capture and decoding utility.
Why it’s special:
Integrated directly into Solaris networking stack (STREAMS-based), faster than early tcpdump.
Linux equivalent:
tcpdump and wireshark perform similar functions now, but snoop was native long before these were standard on UNIX.
Conclusion:
Solaris-originated concept, later paralleled on Linux.

Common or Shared Tools (Not Unique)

These tools or equivalents existed long before Solaris or exist identically in Linux:

Tool	Origin / Availability
`vmstat`, `iostat`, `sar`, `mpstat`, `uptime`	Common to all UNIX systems (from AT&T UNIX / BSD).
`prstat`	Solaris version of `top`. Linux equivalent: `top`, `htop`.
`netstat`, `ping`, `traceroute`	Common across all UNIX variants.
`df`, `du`, `ps`, `dmesg`, `swap`	Standard UNIX commands.

Summary Table

Solaris Tool/Feature	Unique to Solaris?	Linux Equivalent / Derivative
DTrace	Yes	`SystemTap`, `eBPF`, `bpftrace`
FMA (fmadm, fmstat)	Yes	Partial RAS tools
SMF (svcs, svcadm)	Yes	`systemd` (inspired by SMF)
ZFS (zpool, zfs)	Yes	OpenZFS (ported)
kstat	Yes	`/proc`, `/sys` (conceptually similar)
snoop	Originally Solaris	`tcpdump` later filled the role
kvmstat, cpustat, psrinfo, pbind	Originally Solaris	`perf`, `taskset`, `numactl` (similar idea)
vmstat, iostat, sar, prstat, mpstat, netstat	Common UNIX heritage	Shared across all UNIX/Linux systems

In summary:

Solaris-original innovations: DTrace, FMA, SMF, ZFS, kstat
Later imitated or ported to Linux: systemd, eBPF, OpenZFS, bpftrace
Traditional UNIX tools: shared across all UNIX flavors.

Here’s a comprehensive comparison chart between Solaris and Linux system performance/monitoring tools, including what each does, how to use it, and the closest equivalent command or concept in Linux.

Solaris vs Linux System Performance and Monitoring Tools

Category	Solaris Tool	Linux Equivalent	Purpose / Notes
CPU Monitoring	`prstat`	`top`, `htop`	Displays real-time CPU and memory usage per process or per user.
	`mpstat`	`mpstat` (sysstat package)	CPU usage per processor/core. Syntax is almost identical.
	`psrinfo`	`lscpu`	Displays processor status, speed, and count.
	`psradm`	`echo 0 > /sys/devices/system/cpu/cpuX/online`	Enables/disables CPUs. Solaris command is more straightforward.
	`cpustat`	`perf stat`, `perf top`	Collects CPU hardware counter statistics.
	`pbind`	`taskset`	Binds a process to a specific CPU. Used for CPU affinity.

| Comprehensive Tracing & Diagnostics | DTrace | SystemTap, perf, ftrace, eBPF, bpftrace | Solaris’s most famous tool — dynamic kernel/user tracing. Linux tools were all later attempts to replicate its power. |
| | kstat | /proc, /sys, sysstat tools | Solaris kernel statistics interface; Linux uses virtual filesystems for similar info. |

Summary by Category

Area	Solaris Original or Superior Tool	Linux Later Counterpart
Deep Tracing	DTrace	`eBPF`, `bpftrace`, `SystemTap`
Service Management	SMF (svcs/svcadm)	`systemd`
Filesystem	ZFS	`OpenZFS` (ported)
Fault Management	FMA (fmadm/fmstat)	Partial: `mcelog`, `rasdaemon`
Kernel Stats Interface	kstat	`/proc`, `/sys`
Networking Sniffer	snoop	`tcpdump`
Containerization	Zones	`LXC`, `Docker`
Common UNIX Tools	`iostat`, `vmstat`, `mpstat`, `sar`	Same tools, shared UNIX heritage

Key Takeaways

Solaris was the innovator — DTrace, ZFS, SMF, and Zones were years ahead of Linux equivalents.
Linux caught up by borrowing the concepts, not the implementations (due to licensing differences).
Shared tools (iostat, vmstat, sar) come from the common UNIX System V heritage.
For deep kernel-level debugging and self-healing, Solaris still has unmatched elegance (especially with DTrace + FMA).

BLU // SAS