【问题标题】:Why does host_statistics64() return inconsistent results?为什么 host_statistics64() 返回不一致的结果?
【发布时间】:2013-02-09 15:37:36
【问题描述】:

为什么 OS X 10.6.8 中的 host_statistics64()(我不知道其他版本是否有这个问题)返回的免费、活动、非活动和有线内存的计数加起来不等于总数内存?为什么它缺少不一致的页数?

以下输出表示十秒内未分类为空闲、活动、非活动或有线的页面数(大约每秒采样一次)。

458
243
153
199
357
140
304
93
181
224

产生上述数字的代码是:

#include <stdio.h>
#include <mach/mach.h>
#include <mach/vm_statistics.h>
#include <sys/types.h>
#include <sys/sysctl.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char** argv) {
        struct vm_statistics64 stats;
        mach_port_t host    = mach_host_self();
        natural_t   count   = HOST_VM_INFO64_COUNT;
        natural_t   missing = 0;
        int         debug   = argc == 2 ? !strcmp(argv[1], "-v") : 0;
        kern_return_t ret;
        int           mib[2];
        long          ram;
        natural_t     pages;
        size_t        length;
        int           i;

        mib[0] = CTL_HW;
        mib[1] = HW_MEMSIZE;
        length = sizeof(long);
        sysctl(mib, 2, &ram, &length, NULL, 0);
        pages  = ram / getpagesize();

        for (i = 0; i < 10; i++) {
                if ((ret = host_statistics64(host, HOST_VM_INFO64, (host_info64_t)&stats, &count)) != KERN_SUCCESS) {
                        printf("oops\n");
                        return 1;
                }

                /* updated for 10.9 */
                missing = pages - (
                        stats.free_count     +
                        stats.active_count   +
                        stats.inactive_count +
                        stats.wire_count     +
                        stats.compressor_page_count
                );

                if (debug) {
                        printf(
                                "%11d pages (# of pages)\n"
                                "%11d free_count (# of pages free) \n"
                                "%11d active_count (# of pages active) \n"
                                "%11d inactive_count (# of pages inactive) \n"
                                "%11d wire_count (# of pages wired down) \n"
                                "%11lld zero_fill_count (# of zero fill pages) \n"
                                "%11lld reactivations (# of pages reactivated) \n"
                                "%11lld pageins (# of pageins) \n"
                                "%11lld pageouts (# of pageouts) \n"
                                "%11lld faults (# of faults) \n"
                                "%11lld cow_faults (# of copy-on-writes) \n"
                                "%11lld lookups (object cache lookups) \n"
                                "%11lld hits (object cache hits) \n"
                                "%11lld purges (# of pages purged) \n"
                                "%11d purgeable_count (# of pages purgeable) \n"
                                "%11d speculative_count (# of pages speculative (also counted in free_count)) \n"
                                "%11lld decompressions (# of pages decompressed) \n"
                                "%11lld compressions (# of pages compressed) \n"
                                "%11lld swapins (# of pages swapped in (via compression segments)) \n"
                                "%11lld swapouts (# of pages swapped out (via compression segments)) \n"
                                "%11d compressor_page_count (# of pages used by the compressed pager to hold all the compressed data) \n"
                                "%11d throttled_count (# of pages throttled) \n"
                                "%11d external_page_count (# of pages that are file-backed (non-swap)) \n"
                                "%11d internal_page_count (# of pages that are anonymous) \n"
                                "%11lld total_uncompressed_pages_in_compressor (# of pages (uncompressed) held within the compressor.) \n",
                                pages, stats.free_count, stats.active_count, stats.inactive_count,
                                stats.wire_count, stats.zero_fill_count, stats.reactivations,
                                stats.pageins, stats.pageouts, stats.faults, stats.cow_faults,
                                stats.lookups, stats.hits, stats.purges, stats.purgeable_count,
                                stats.speculative_count, stats.decompressions, stats.compressions,
                                stats.swapins, stats.swapouts, stats.compressor_page_count,
                                stats.throttled_count, stats.external_page_count,
                                stats.internal_page_count, stats.total_uncompressed_pages_in_compressor
                        );
                }

                printf("%i\n", missing);
                sleep(1);
        }

        return 0;
}

【问题讨论】:

  • vm_stat,使用相同的代码,有同样的不完全加法问题。所以,这不是你的代码本身的问题。
  • 使用 ML 和 8 GB RAM,缺少的部分更大(大约 2400)。
  • 看看opensource.apple.com/source/top/top-73/libtop.c,里面好像有一些特殊的计算。
  • @nneonneo 哦,我知道。我试图弄清楚为什么 vm_stat 给我的数据不好。
  • @patrix 我不知道这是否是巧合,但这似乎是正确的。我最糟糕的情况是丢失了 500 到 600 页,而我有两个 CPU。如果我们将其扩展到 8 个 CPU,您会看到 2000 到 2400 个页面丢失。如果是这种情况,那么这些页面当前没有被任何 CPU 声明;这意味着它们很可能正在 CPU 之间传输。

标签: macos memory darwin mach


【解决方案1】:

TL;DR:

  • host_statistics64() 从不同来源获取信息,这可能会花费时间并可能产生不一致的结果。
  • host_statistics64() 通过名称如vm_page_foo_count 的变量获取一些信息。但并非所有这些变量都被考虑在内,例如vm_page_stolen_count 不是。
  • 众所周知的/usr/bin/top被盗页面添加到有线页面的数量中。这表明在计算页数时应考虑这些页数。

备注

  • 我正在使用带有Darwin Kernel Version 16.5.0 xnu-3789.51.2~3/RELEASE_X86_64 x86_64 的 macOS 10.12,但所有行为都是完全可重现的。
  • 我将链接很多我在我的机器上使用的 XNU 版本的源代码。可以在这里找到:xnu-3789.51.2
  • 您编写的程序与/usr/bin/vm_stat 基本相同,只是host_statistics64()(和host_statistics())的包装。对应的源代码可以在这里找到:system_cmds-496/vm_stat.tproj/vm_stat.c

host_statistics64() 如何融入 XNU 以及它是如何工作的?

widley 知道 OS X 内核被称为 XNU (XNU IS NOT UNIX) “是一个混合内核,将卡内基梅隆大学开发的 Mach 内核与来自 FreeBSD 和 C++ API 的组件相结合,用于编写称为 IOKit 的驱动程序。” (https://github.com/opensource-apple/xnu/blob/10.12/README.md)

虚拟内存管理 (VM) 是 Mach 的一部分,因此host_statistics64() 位于此处。让我们仔细看看它的实现,它包含在xnu-3789.51.2/osfmk/kern/host.c 中。

函数签名是

kern_return_t
host_statistics64(host_t host, host_flavor_t flavor, host_info64_t info, mach_msg_type_number_t * count);

第一个相关的行是

[...]
processor_t processor;
vm_statistics64_t stat;
vm_statistics64_data_t host_vm_stat;
mach_msg_type_number_t original_count;
unsigned int local_q_internal_count;
unsigned int local_q_external_count;
[...]
processor = processor_list;
stat = &PROCESSOR_DATA(processor, vm_stat);
host_vm_stat = *stat;

if (processor_count > 1) {
    simple_lock(&processor_list_lock);

    while ((processor = processor->processor_list) != NULL) {
        stat = &PROCESSOR_DATA(processor, vm_stat);

        host_vm_stat.zero_fill_count += stat->zero_fill_count;
        host_vm_stat.reactivations += stat->reactivations;
        host_vm_stat.pageins += stat->pageins;
        host_vm_stat.pageouts += stat->pageouts;
        host_vm_stat.faults += stat->faults;
        host_vm_stat.cow_faults += stat->cow_faults;
        host_vm_stat.lookups += stat->lookups;
        host_vm_stat.hits += stat->hits;
        host_vm_stat.compressions += stat->compressions;
        host_vm_stat.decompressions += stat->decompressions;
        host_vm_stat.swapins += stat->swapins;
        host_vm_stat.swapouts += stat->swapouts;
    }

    simple_unlock(&processor_list_lock);
}
[...]

我们得到host_vm_stat,它的类型是vm_statistics64_data_t。这只是一个typedef struct vm_statistics64,您可以在xnu-3789.51.2/osfmk/mach/vm_statistics.h 中看到。我们从 xnu-3789.51.2/osfmk/kern/processor_data.h 中定义的 makro PROCESSOR_DATA() 获取处理器信息。我们在循环遍历所有处理器时填充host_vm_stat,只需将相关数字相加即可。

如您所见,我们发现了一些众所周知的统计数据,例如 zero_fill_countcompressions,但host_statistics64() 并未涵盖所有数据。

接下来的相关行是:

stat = (vm_statistics64_t)info;

stat->free_count = vm_page_free_count + vm_page_speculative_count;
stat->active_count = vm_page_active_count;
[...]
stat->inactive_count = vm_page_inactive_count;
stat->wire_count = vm_page_wire_count + vm_page_throttled_count + vm_lopage_free_count;
stat->zero_fill_count = host_vm_stat.zero_fill_count;
stat->reactivations = host_vm_stat.reactivations;
stat->pageins = host_vm_stat.pageins;
stat->pageouts = host_vm_stat.pageouts;
stat->faults = host_vm_stat.faults;
stat->cow_faults = host_vm_stat.cow_faults;
stat->lookups = host_vm_stat.lookups;
stat->hits = host_vm_stat.hits;

stat->purgeable_count = vm_page_purgeable_count;
stat->purges = vm_page_purged_count;

stat->speculative_count = vm_page_speculative_count;

我们重用stat 并将其作为我们的输出结构。然后我们用两个unsigned long 的总和填充free_count,称为vm_page_free_countvm_page_speculative_count。我们以相同的方式收集其他剩余数据(通过使用名为 vm_page_foo_count 的变量)或从上面填写的 host_vm_stat 中获取统计信息。

1.结论 我们从不同来源收集数据。来自处理器信息或来自名为vm_page_foo_count 的变量。这会花费时间,并且可能会导致一些不一致的问题,因为 VM 是一个非常快速且连续的过程。

让我们仔细看看已经提到的变量vm_page_foo_count。它们在xnu-3789.51.2/osfmk/vm/vm_page.h 中定义如下:

extern
unsigned int    vm_page_free_count; /* How many pages are free? (sum of all colors) */
extern
unsigned int    vm_page_active_count;   /* How many pages are active? */
extern
unsigned int    vm_page_inactive_count; /* How many pages are inactive? */
#if CONFIG_SECLUDED_MEMORY
extern
unsigned int    vm_page_secluded_count; /* How many pages are secluded? */
extern
unsigned int    vm_page_secluded_count_free;
extern
unsigned int    vm_page_secluded_count_inuse;
#endif /* CONFIG_SECLUDED_MEMORY */
extern
unsigned int    vm_page_cleaned_count; /* How many pages are in the clean queue? */
extern
unsigned int    vm_page_throttled_count;/* How many inactives are throttled */
extern
unsigned int    vm_page_speculative_count;  /* How many speculative pages are unclaimed? */
extern unsigned int vm_page_pageable_internal_count;
extern unsigned int vm_page_pageable_external_count;
extern
unsigned int    vm_page_xpmapped_external_count;    /* How many pages are mapped executable? */
extern
unsigned int    vm_page_external_count; /* How many pages are file-backed? */
extern
unsigned int    vm_page_internal_count; /* How many pages are anonymous? */
extern
unsigned int    vm_page_wire_count;     /* How many pages are wired? */
extern
unsigned int    vm_page_wire_count_initial; /* How many pages wired at startup */
extern
unsigned int    vm_page_free_target;    /* How many do we want free? */
extern
unsigned int    vm_page_free_min;   /* When to wakeup pageout */
extern
unsigned int    vm_page_throttle_limit; /* When to throttle new page creation */
extern
uint32_t    vm_page_creation_throttle;  /* When to throttle new page creation */
extern
unsigned int    vm_page_inactive_target;/* How many do we want inactive? */
#if CONFIG_SECLUDED_MEMORY
extern
unsigned int    vm_page_secluded_target;/* How many do we want secluded? */
#endif /* CONFIG_SECLUDED_MEMORY */
extern
unsigned int    vm_page_anonymous_min;  /* When it's ok to pre-clean */
extern
unsigned int    vm_page_inactive_min;   /* When to wakeup pageout */
extern
unsigned int    vm_page_free_reserved;  /* How many pages reserved to do pageout */
extern
unsigned int    vm_page_throttle_count; /* Count of page allocations throttled */
extern
unsigned int    vm_page_gobble_count;
extern
unsigned int    vm_page_stolen_count;   /* Count of stolen pages not acccounted in zones */
[...]
extern
unsigned int    vm_page_purgeable_count;/* How many pages are purgeable now ? */
extern
unsigned int    vm_page_purgeable_wired_count;/* How many purgeable pages are wired now ? */
extern
uint64_t    vm_page_purged_count;   /* How many pages got purged so far ? */

关于我们使用host_statistics64() 只能访问非常有限的数量的大量统计数据。这些统计数据中的大部分都在xnu-3789.51.2/osfmk/vm/vm_resident.c 中更新。例如此函数将页面释放到空闲页面列表中:

/*
*   vm_page_release:
*
*   Return a page to the free list.
*/

void
vm_page_release(
    vm_page_t   mem,
    boolean_t   page_queues_locked)
{
    [...]
    vm_page_free_count++;
    [...]
}

非常有趣的是extern unsigned int vm_page_stolen_count; /* Count of stolen pages not acccounted in zones */。什么是被盗页面?似乎有一些机制可以从某些列表中取出一个页面,即使它通常不会被分页。其中一种机制是推测页面列表中页面的agexnu-3789.51.2/osfmk/vm/vm_page.h告诉我们

* VM_PAGE_MAX_SPECULATIVE_AGE_Q * VM_PAGE_SPECULATIVE_Q_AGE_MS
* defines the amount of time a speculative page is normally
* allowed to live in the 'protected' state (i.e. not available
* to be stolen if vm_pageout_scan is running and looking for
* pages)...  however, if the total number of speculative pages
* in the protected state exceeds our limit (defined in vm_pageout.c)
* and there are none available in VM_PAGE_SPECULATIVE_AGED_Q, then
* vm_pageout_scan is allowed to steal pages from the protected
* bucket even if they are underage.
*
* vm_pageout_scan is also allowed to pull pages from a protected
* bin if the bin has reached the "age of consent" we've set

确实是void vm_pageout_scan(void) 增加了vm_page_stolen_count。你可以在xnu-3789.51.2/osfmk/vm/vm_pageout.c找到对应的源码。

我认为host_statistics64() 计算 VM 统计信息时不会考虑被盗页面。

证明我是对的

证明这一点的最佳方法是使用自定义版本的host_statistics64() 手动编译 XNU。我没有机会这样做,但很快就会尝试。

幸运的是,我们并不是唯一对正确的 VM 统计数据感兴趣的人。因此,我们应该看看众所周知的/usr/bin/top(XNU 中不包含)的实现,它在此处完全可用:top-108(我刚刚选择了macOS 10.12.4 release)。

让我们看看top-108/libtop.c,我们在其中找到以下内容:

static int
libtop_tsamp_update_vm_stats(libtop_tsamp_t* tsamp) {
    kern_return_t kr;
    tsamp->p_vm_stat = tsamp->vm_stat;

    mach_msg_type_number_t count = sizeof(tsamp->vm_stat) / sizeof(natural_t);
    kr = host_statistics64(libtop_port, HOST_VM_INFO64, (host_info64_t)&tsamp->vm_stat, &count);
    if (kr != KERN_SUCCESS) {
        return kr;
    }

    if (tsamp->pages_stolen > 0) {
        tsamp->vm_stat.wire_count += tsamp->pages_stolen;
    }

    [...]

    return kr;
}

tsamplibtop_tsamp_t 类型,它是在top-108/libtop.h 中定义的结构。它包含 vm_statistics64_data_t vm_statuint64_t pages_stolen 等内容。

如您所见,static int libtop_tsamp_update_vm_stats(libtop_tsamp_t* tsamp) 得到tsamp-&gt;vm_stathost_statistics64() 填充,正如我们所知。之后它检查是否tsamp-&gt;pages_stolen &gt; 0 并将其添加到tsamp-&gt;vm_statwire_count 字段中。

2.结论 如果我们只使用host_statistics64() (如/usr/bin/vm_stat 或您的示例代码),我们将无法获得这些被盗页面的数量!

为什么host_statistics64() 按原样实施?

老实说,我不知道。分页是一个复杂的过程,因此实时观察是一项具有挑战性的任务。我们必须注意到,它的实现似乎没有错误。我认为如果我们可以访问vm_page_stolen_count,我们甚至不会获得 100% 准确的页数。 /usr/bin/top 的实现如果数量不是很大,则不计算被盗页面。

另一个有趣的事情是函数static void update_pages_stolen(libtop_tsamp_t *tsamp) 上方的注释,即/* This is for &lt;rdar://problem/6410098&gt;. */Open Radar 是 Apple 软件的错误报告网站,通常以评论中给出的格式对错误进行分类。我无法找到相关的错误;可能是因为缺页。

希望这些信息对您有所帮助。如果我设法在我的机器上编译最新(和定制)版本的 XNU,我会告诉你的。也许这会带来有趣的见解。

【讨论】:

    【解决方案2】:

    刚刚注意到,如果您将 compressor_page_count 添加到混合中,您会更接近机器中的实际 RAM 数量。

    这是一个观察,而不是解释,如果有正确记录的链接,那就太好了!

    【讨论】:

    • 压缩页面出现在 10.9,正如问题所述,这是 10.6。我已经更新了 10.9 的代码,但它仍然缺少与 10.6 下大致相同数量的 10.9 下的页面(带有添加的字段)。
    猜你喜欢
    • 2021-04-20
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2016-03-10
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多