【问题标题】:I've been trying to match the a name received from 2 sources with each other and check if they are almost a match or not我一直在尝试将从 2 个来源收到的名称相互匹配,并检查它们是否几乎匹配
【发布时间】:2019-07-18 10:29:07
【问题描述】:

在示例数据中,我列出了我们从 2 个不同来源收到的特定人员(潜在客户)的雇主姓名。 我一直在尝试找到一种方法来更好地匹配这两个名称并获得良好的结果。 (目前,它是作为手动工作完成的) 我不认为我在尝试做不可能的事情......但如果它无法实现,请不要苛刻!

以下是根据手动验证“匹配”的数据集。

        ADDUS==============================================Addus Home Care
        Amazon.com, Inc. and its affiliates=====================Amazon.com
        Aon========================================Aon Service Corporation
        ARAMARK Food & Support Svc.================================Aramark
        AT&T Mobility Services LLC===========================AT&T Mobility
        CDW, LLC===========================================CDW Corporation
        Lurie Children's Hospital of Chicago======Lurie Childrens Hospital
        Securitas Security Services USA, Inc============Securitas security
        The PNC Financial Services Group, Inc.======================PNC NA
        United States Department of Homeland Security====US Homeland Securiti
        TCS=========================================Tata Consultancy Services

虽然几乎是显而易见的,但为了强调,让我陈述一下。

  1. 来自这些来源的名称可能存在拼写错误
  2. 可能有缩写(例如:TCS 在一个地方,Tata Consultancy 在另一个地方)

请向我推荐一种算法或方法,以最少的“错误接受案例”数量来做到这一点 - 我的意思是这样的案例,它们从不同的算法中获得了很高的匹配率。

请尝试提出一种方法。

【问题讨论】:

  • 好吧,我想我可以从 stackoverflow 获得一两个想法。但我想没有!如果有任何人可以提供任何意见,请提供。

标签: string-comparison similarity levenshtein-distance fuzzy-logic edit-distance


【解决方案1】:

我只看到一个,但随着时间的推移,这是一个相当进步和准确的选项: (1) 首先要注意:你有你的“体力工作”,你会坚持下去。 (2) 但现在更好的部分是:随着时间的推移,您分类的数据越多,手动工作变得越来越短 - 一种自学机器。看下面的尝试说明,有兴趣的我们可以稍后再讨论。

1. Yur current workflow
    1. create a initial employer list of triplets.
        1. employer1 (string)
        2. employer2 (string)
        3. equivalence (values {VALID|INVALID}), default: INVALID

       Result: AllEpmployersList, unverified.

    2. Process the AllEpmployersList manually
        1. for each AllEpmployersList member (triplet) 
            1. set the value for equivalence element
               VALID or INVALID respectively.

       Result: VerifiedEpmployersList, triplets  with verified equivalence value.

    3. Use the VerifiedEpmployersList as required for downstream processing.

2. The Adapted (advanced) new workflow
    1. create a initial employer list of triplets.
        1. employer1 (string)
        2. employer2 (string)
        3. equivalence (values {VALID|INVALID}), default: INVALID

       Result: AllEpmployersList, unverified.

    2. feed unverified AllEpmployersList into matchKnownEmployers process (described later).

       Result: two lists, AllKnownEmployers and AllUnknownEployers.

    3. Process the AllUnknownEployers list manually.
       Result: VerifiedEpmployersList with verified equivalence value.

    4. feed the VerifiedEpmployersList list into importKnownEmployers process

    5. feed (again) the AllEpmployerList (Result 2.1) into matchKnownEmployers process

       Result:two lists, AllKnownEmployers and AllUnknownEployers.

    6. Use the AllKnownEmployers as required for downstream processes.


3. Required Investments (instances you have to establish)
    1. create KnownEmployers database
        1. create table knownEmployerNames,
            1. columns:
                1. id
                2. employerName
                3. aliasIdValue
        2. create table lastAliasIdValue
            1. columns:
                1. aliasIdValue
        3. init table lastAliasIdValue
            1. insert one initial row, aliasIdValue = 0

    2. create matchKnownEmployersProcess with this characteristics:
        1. Input data:  employerList (triplets)
        2. init empty list for knownEmployers and unknownEployers
        3. for each member in employerList do:
            1. if employer1 and  employer2 in table knownEmployerNames and employer1::aliasIdValue equals employer2::aliasIdValue
                1. then set member::equivalence value to VALID and append the member into knownEmployers list
                2. else  append the member  into  unknownEployers list
        4. Output data: two lists, knownEmployers and unknownEployers.

    3. create importKnownEmployersProcess with this characteristics:
        1. Input data: employerList (triplets)
        2. for each element in employerList do:
            1. if equivalence element value is VALID
                1. insert new pattern
                    1. if employer1 or employer2 is in table knownEmployerNames
                        1. then 
                            1. function isUnknown(employer1, employer2) {
            retVal = {}
            retVal[‘aliasIdValue’] = 
                employer1::aliasIdValue ||
                employer2::aliasIdValue
            retVal[‘newEmployer’]  = 
                 (!employer1 || !employer2)
            return retVal
                           }
                            2. aliasIdValue, newEmployer = isUnknown(employer1,  employer2)
                            3. insert aliasIdValue, newEmployer into knownEmployerNames table
                        2. else 
                            1. fetch and increment aliasIdValue from lastAliasIdValue table
                            2. insert into knownEmployerNames (employer1, aliasIdValue) and (employer2, aliasIdValue)
                            3. update incremented lastAliasIdValue in the  lastAliasIdValue table
        3. Output data: none

【讨论】:

    猜你喜欢
    • 2011-03-05
    • 2022-01-25
    • 1970-01-01
    • 2017-11-30
    • 1970-01-01
    • 2019-01-11
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多