如何将计数数据帧转换为概率密度函数

Question

假设我对整数有以下观察：

df = pd.DataFrame({'observed_scores': [100, 100, 90, 85, 100, ...]})

我知道这可以作为输入来制作密度图：

df['observed_scores'].plot.density()

但假设我有一个计数表：

df = pd.DataFrame({'observed_scores': [100, 95, 90, 85, ...], 'counts': [1534, 1399, 3421, 8764, ...})

比完整的observed_scores 系列更便宜（我有很多观察）。

我知道可以使用计数绘制直方图，但是如何绘制密度图？如果可能的话，是否可以在不必将计数表拆分/解开成数千行的情况下完成？

Answer 1

IIUC，statsmodels 可让您拟合加权 KDE：

from statsmodels.nonparametric.kde import KDEUnivariate

df = pd.DataFrame({'observed_scores': [100, 95, 90, 85],
                   'counts': [1534, 1399, 3421, 8764]})

kde1= KDEUnivariate(df.observed_scores)
kde_noweight = KDEUnivariate(df.observed_scores)
kde1.fit(weights=df.counts, fft=False)
kde_noweight.fit()
plt.plot(kde1.support, kde1.density)
plt.plot(kde_noweight.support, kde_noweight.density)
plt.legend(['weighted', 'unweighted'])

输出：

Answer 2

您可以使用 scipy 手动构造加权 kde。只要您将bw_method 指定为标量，这将完美匹配。允许默认时，拟合变得不一致

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

np.random.seed(410112)
# Real counts data to check with
df = pd.DataFrame({'observed_scores': np.random.randint(1, 100, 40000)})

# Aggregated to value_counts, which is what you have access to
df1 = df.groupby('observed_scores').size().to_frame('counts').reset_index()

---

代码

def weighted_kde(y, weights, bw_method=None):
    sample_range = np.nanmax(y) - np.nanmin(y)
    ind = np.linspace(
        np.nanmin(y) - 0.5 * sample_range,
        np.nanmax(y) + 0.5 * sample_range,
        1000)

    gkde = stats.gaussian_kde(y, bw_method=bw_method, weights=weights)

    y = gkde.evaluate(ind)
    
    return ind, y

---

检查输出

bw_method=0.5

fig, ax = plt.subplots()
# Underlying data, with pandas kde
df['observed_scores'].plot.density(ax=ax, bw_method=bw_method, label='pandas density', lw=2)

# From aggregated counts data
ind, y = weighted_kde(df1['observed_scores'], df1['counts'], bw_method=bw_method)
ax.plot(ind, y, label='Manual Weighted KDE', lw=2, linestyle='--')

ax.legend()
plt.show()