Baeldung Pro – Linux – NPI EA (cat = Baeldung on Linux)
announcement - icon

It's finally here:

>> The Road to Membership and Baeldung Pro.

Going into ads, no-ads reading, and bit about how Baeldung works if you're curious :)

Partner – Orkes – NPI EA (tag=Kubernetes)
announcement - icon

Modern software architecture is often broken. Slow delivery leads to missed opportunities, innovation is stalled due to architectural complexities, and engineering resources are exceedingly expensive.

Orkes is the leading workflow orchestration platform built to enable teams to transform the way they develop, connect, and deploy applications, microservices, AI agents, and more.

With Orkes Conductor managed through Orkes Cloud, developers can focus on building mission critical applications without worrying about infrastructure maintenance to meet goals and, simply put, taking new products live faster and reducing total cost of ownership.

Try a 14-Day Free Trial of Orkes Conductor today.

1. Overview

In this tutorial, we’ll talk about memory usage analysis in Linux. First, we’ll discuss how Linux applications work and how they share memory among them to optimize resources. This affects the definition of memory usage for a single application.

Then, we’ll discuss ways to inspect the memory usage of a given application. We’ll be able to distinguish which part of the memory usage is exclusive to an application versus shared with others.

2. Understanding Memory in Linux

The first thing we need to discuss is what we refer to as “memory usage”. Most applications in Linux use shared libraries. Applications require these libraries to run, but other applications can also use these libraries.

This means that it’s not straightforward to state the exact memory usage for a given application that uses shared libraries. Should we count the memory used by the application without the shared libraries? We can argue that the application requires these libraries to run. However, counting the total memory used by the application and the shared libraries will include some shared libraries that Linux loads by default.

2.1. Potentially Erroneous Usage of ps

Let’s think about the ps command. This command helps us to monitor processes in our system. So, it’s intuitive to use it to explore the memory usage of applications in our system.

However, we should be careful when interpreting the memory information that ps gives. The top command may also present values that are not what we are after.

We’ll see this by running the sleep command, detaching it from the terminal, and inspecting the PID of the process with ps:

$ sleep 15 &
[1] 611040
$ ps -u -p 611040
USER           PID    %CPU    %MEM       VSZ      RSS    TTY        STAT   START    TIME   COMMAND
username    611040     0.0     0.0      5544     2076    pts/3      S      01:45    0:00   sleep 15

We’ve used the -u flag to request information on the process’s user. We’ve also got several entries with extra information.

The entries related to the memory usage are VSZ (Virtual Set Size) and RSS (Resident Set Size). The values are reported in kilobytes, which means that our sleep command is taking 5544 Kb in VSZ and 2076 Kb in RSS. But, which one represents the memory that the sleep command is using?

VSZ indicates the maximum RAM memory that a process can access (including both the memory in use, the memory that is not in use but that the process may use, and the shared libraries). Whereas, RSS only counts the RAM in use but not the shared libraries loaded in memory. The output of ps is not wrong, but we must know how to interpret the values we get.

Several applications may be sharing the same library. This means that the VSZ is giving too large of a value. The VSZ would be an accurate memory usage metric if the machine was running this as the only process. Using the RSS is not accurate since it doesn’t even account for shared libraries.

3. Using pmap to Get Accurate Memory Information

The pmap command reports the memory map of a process. We can use pmap to get the memory usage of a process and detail the libraries and binary files.

By default, pmap will only show us the processes that we own. If we want to see information of all processes, we’ll need to run it as a super-user.

Let’s run pmap with the -d flag and with the PID of the process we want to inspect:

$ sudo pmap -d 611040
611040:   sleep 15
Address           Kbytes Mode  Offset           Device    Mapping
000061fabc20c000       8 r---- 0000000000000000 008:00007 sleep
000061fabc20e000      16 r-x-- 0000000000002000 008:00007 sleep
000061fabc212000       4 r---- 0000000000006000 008:00007 sleep
000061fabc213000       4 r---- 0000000000007000 008:00007 sleep
000061fabc214000       4 rw--- 0000000000008000 008:00007 sleep
000061faf443c000     132 rw--- 0000000000000000 000:00000   [ anon ]
000070541c800000    2988 r---- 0000000000000000 008:00007 locale-archive
000070541cb21000      12 rw--- 0000000000000000 000:00000   [ anon ]
000070541cb24000     144 r---- 0000000000000000 008:00007 libc.so.6
000070541cb48000    1476 r-x-- 0000000000024000 008:00007 libc.so.6
000070541ccb9000     312 r---- 0000000000195000 008:00007 libc.so.6
000070541cd07000      16 r---- 00000000001e3000 008:00007 libc.so.6
000070541cd0b000       8 rw--- 00000000001e7000 008:00007 libc.so.6
000070541cd0d000      32 rw--- 0000000000000000 000:00000   [ anon ]
000070541cd2e000       8 rw--- 0000000000000000 000:00000   [ anon ]
000070541cd30000      16 r---- 0000000000000000 000:00000   [ anon ]
000070541cd34000       8 r-x-- 0000000000000000 000:00000   [ anon ]
000070541cd36000       4 r---- 0000000000000000 008:00007 ld-linux-x86-64.so.2
000070541cd37000     164 r-x-- 0000000000001000 008:00007 ld-linux-x86-64.so.2
000070541cd60000      40 r---- 000000000002a000 008:00007 ld-linux-x86-64.so.2
000070541cd6a000       8 r---- 0000000000034000 008:00007 ld-linux-x86-64.so.2
000070541cd6c000       8 rw--- 0000000000036000 008:00007 ld-linux-x86-64.so.2
00007fffd2074000     132 rw--- 0000000000000000 000:00000   [ stack ]
ffffffffff600000       4 --x-- 0000000000000000 000:00000   [ anon ]
mapped: 5548K    writeable/private: 336K    shared: 0K

We use the -d flag to show the device format and provide an extra summary at the bottom. In this summary at the bottom, we can see the mapped memory. This corresponds to the VSZ in the ps command.

Next to it, we see the writeable/private memory that this process is taking without the shared libraries. The reported value of 336 Kb is considerably different than the 5544 Kb reported by ps!

More importantly, pmap gives us a breakdown of the memory usage by components, something we were not getting with ps. We can see that some libraries (for example ld-linux-x86-64.so.2) are listed several times.

To understand why this is important, recall that shared libraries have two main components: the code part (mode “r-x–“) and the data part (mode “rw—“). The code part of the libraries is usually the largest and it’s the part that processes can share. 

This is specifically valuable with applications that create a lot of child processes. Adding a single child process will add only a small chunk of additional memory since the code part of the libraries is shared. But ps will report this as much more used memory since it also considers the code part inside the total child’s memory usage.

There are other two useful flags to take even more advantage of pmap: the -x and -X flags:

$ sudo pmap -x 611040
611040:   sleep 15
Address           Kbytes     RSS   Dirty Mode  Mapping
0000640bef90a000       8       8       0 r---- sleep
0000640bef90c000      16      16       0 r-x-- sleep
0000640bef910000       4       4       0 r---- sleep
...
ffffffffff600000       4       0       0 --x-- [ anon ]
---------------- ------- ------- ------- 
total kB            5548    1968     108
 
$ sudo pmap -X 611040
611040:   sleep 15
         Address Perm   Offset Device   Inode Size  Rss Pss Pss_Dirty Referenced Anonymous KSM ... Mapping
    640bef90a000 r--p 00000000  08:07 3029989    8    8   8         0          8         0   0 ... sleep
    640bef90c000 r-xp 00002000  08:07 3029989   16   16  16         0         16         0   0 ... sleep
    640bef910000 r--p 00006000  08:07 3029989    4    4   4         0          4         0   0 ... sleep
...
ffffffffff600000 --xp 00000000  00:00       0    4    0   0         0          0         0   0 ... [anon]
                                              ==== ==== === ========= ========== ========= === ... 
                                              5548 1968 166       108       1968       108   0 ... KB 

Note that the two outputs have been trimmed both vertically and horizontally. Still, we can see the amount of information that we get out of pmap. We get the RSS values, as well as dirty, mapped, and swapped memory.

pmap also returns the PSS. The PSS includes the unique memory to the application and the proportional value of the shared memory divided by the number of processes using those shared libraries. It is a good representation of the actual memory usage of an application.

4. Alternatives to pmap

pmap is not the only tool that exists to get detailed information on memory usage. We’ll briefly present other tools in this section.

4.1. Bundled With Linux

We can inspect the /proc pseudo-filesystem to get information related to a process. But as before, we must know what file we check. The information is inside the /proc/<pid> folder.

On the one hand, the status file provides information similar to that of ps (compare the 5544 Kb value for VSZ and 2076 Kb value for RSS reported by ps):

$ cat /proc/611040/status
Name:	sleep
Umask:	0022
State:	S (sleeping)
Tgid:	611040
...
VmPeak:	    5544 kB
VmSize:	    5544 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	    2076 kB
VmRSS:	    2076 kB
RssAnon:	       0 kB
RssFile:	    2068 kB
RssShmem:	       0 kB
VmData:	     204 kB
VmStk:	     132 kB
VmExe:	      16 kB
VmLib:	    1648 kB
VmPTE:	      48 kB
VmSwap:	       0 kB
...

The output has been trimmed since it contains other information apart from memory usage.

On the other hand, the smaps file provides information similar to pmap, including the breakdown by components and other memory metrics:

$ cat /proc/611040/smaps
5f7d1dbf7000-5f7d1dbf9000 r--p 00000000 08:07 3029989                    /usr/bin/sleep
Size:                  8 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                   8 kB
Pss:                   8 kB
Pss_Dirty:             0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         8 kB
Private_Dirty:         0 kB
Referenced:            8 kB
Anonymous:             0 kB
KSM:                   0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:           0
VmFlags: rd mr mw me sd 
5f7d1dbf9000-5f7d1dbfd000 r-xp 00002000 08:07 3029989                    /usr/bin/sleep
Size:                 16 kB
...
5f7d39dbc000-5f7d39ddd000 rw-p 00000000 00:00 0                          [heap]
Size:                132 kB
...
7faabce00000-7faabd0eb000 r--p 00000000 08:07 3030748                    /usr/lib/locale/locale-archive
Size:               2988 kB
...
7faabd1e6000-7faabd20a000 r--p 00000000 08:07 3018050                    /usr/lib/libc.so.6
Size:                144 kB
...
7faabd3f2000-7faabd3f6000 r--p 00000000 00:00 0                          [vvar]
Size:                 16 kB
...
VmFlags: rd ex mr mw me de sd 
7faabd3f8000-7faabd3f9000 r--p 00000000 08:07 3018037                    /usr/lib/ld-linux-x86-64.so.2
Size:                  4 kB
...

This output has also been trimmed, but we can see it follows the same information as pmap. This file is the source of it, and it can be useful for systems that don’t have pmap.

4.2. Other Packages

There are other applications that we can install to get memory usage information. smem is an alternative to ps that returns the VSZ and RSS values, but also values such as the Unique Set Size (USS) and the Proportional Set Size (PSS). The USS value is the amount of memory unique to the process, without including the shared memory.

To analyze the memory usage of the whole system and that of a single application (including heap and shared libraries), we can use exmap. exmap will find the processes that consume the most memory, those with the most writable usage, check memory leaks, consider memory usage inefficiencies, and more. It’s a powerful tool that is worth knowing about.

Finally, we can also consider valgrind (combined with alleyoop to visualize and interpret the results). valgrind serves to debug the memory usage of applications, detect memory leaks, and profile applications. Using valgrind returns very detailed information but we must know that it affects the timings of the application, slowing it down and changing its behavior.

5. Conclusion

In this article, we talked about memory usage in Linux. We discussed memory sharing between applications in Linux. The way memory works impacts our definition of memory usage.

We also discussed several tools available to inspect the memory usage in Linux. We saw that tools as ps return values that may be misleading if we don’t know how to interpret them correctly. Other tools, such as pmap give more detailed information and a breakdown of the memory usage to get better insights.

Finally, we presented other ways and tools to access memory information if pmap is unavailable or we want other usage data.