从文本中获取书目列表及其计数 - Python

Question

在我的 python 任务中，我必须阅读 PDF 论文并获取所有参考文献及其计数（在论文中提到）。 This is the PDF as example 它有 18 个参考文献，并且说 Ref#1 在论文中被提到了 3 次，而 Ref#2 被提到了 1 次，所以这就是我想要的；

Ref#  Count   Reference 
 1     3      Arto Anttila. 1995. How to recognise subjects in English. In Karlsson et al., chapt. 9, pp. 315-358.
 2      1     Dekang Lin. 1996. Evaluation of Principar with the Susanne corpus. In John arroll, editor, Workshop on Robust Parsing, pages 54-69, Prague
 ...

我已经完成了列表中的 Ref # 和 References，并且设法通过使用此正则表达式从包含 Reference 的文本中获取行：

regex = re.compile(r'[A-Z]{1}[a-z\u0000-\u007F]+ \([0-9]{4}\)|\([A-Z]{1}[a-z\u0000-\u007F]+, [0-9]{4}\)|\([A-Z]{1}[a-z\u0000-\u007F]+, [0-9]{4}; [A-Za-z \u0000-\u007F,;]*\)|[A-Z]{1}[a-z\u0000-\u007F]+ \([0-9]{4},[A-Za-z0-9\u0000-\u007F ]*\)|[A-Z]{1}[a-z\u0000-\u007F ]+ [a-z]{2} [a-z]{2}. \([0-9]{4}\)')

所以当我遍历字符串列表（由句子分割的文本）并使用上面的正则表达式查找时：

for i in range(0, len(lstString)):
    refLine = re.findall(regex, lstString[i])
    if(refLine != [] and refLine [0] != []):
        print(refLine)

我得到一些这样的输出：

    (Karls- son et al., 1995)
    Our work is partly based on the work done with the Constraint Grammar framework that was orig- inally proposed by Fred Karlsson
(1990)
    (Tapanainen, 1996)
    (Tapanainen, 1996) is dif- ferent from the former (Karlsson et al., 1995)
    Hurskainen (1996)
    In essence, the same formalism is used in the syn- tactic analysis in J~rvinen (1994) and     Anttila (1995)
    Our notation follows the classical model of depen- dency theory (Heringer, 1993) introduced by Lucien Tesni~re (1959) and later
advocated by Igor Mel'~uk (1987)
    Hudson (1991)
    (Hays, 1964)
    (McCord, 1990; Sleator and Tem- perley, 1991; Eisner, 1996)
    (Hudson, 1991)
    (J~irvinen, 1994)
    The CG-2 program (Tapanainen, 1996) runs a mod- ified disambiguation grammar of Voutilainen (1995)
    (J~rvinen, 1994; Tapanainen and J/~rvinen, 1994)
    (Eisner, 1996)
    Dekang Lin (1996)
    Acknowledgments We are using Atro Voutilainen's (1995)

它返回所有包含引用的字符串，但我遇到了一些问题，例如

1. 它没有像这样捕获参考 Karlsson 等人。 (1995)
2. 其中一些包含 2 个参考
3. 如何更新参考列表中每个参考的计数

我尝试了这段代码来获取每个 ref 的计数，但它总是返回整个列表；

matching = [s for s in lstRef if any(xs in s for xs in refLine)]

我们将不胜感激。

Answer 1

我想知道如果在文档末尾从References 获取姓名（和年份）并使用它们来搜索文档中的引用。

在上一个问题中，您会得到在文档末尾获得References 的代码。

使用正则表达式 '((.*)\. (\d{4})\. 我可以将名称作为一个字符串，将年份作为一个字符串（最终两者都在一个字符串中）

    authors_and_year = re.match('((.*)\. (\d{4})\.)', line)
    text, authors, year = authors_and_year.groups()

即。

   text: Christer Samuelsson, Pasi Tapanainen, and Atro Voutilainen. 1996.
authors: Christer Samuelsson, Pasi Tapanainen, and Atro Voutilainen
   year: 1996

使用下一个正则表达式 ',[ ]*and |,[ ]*| and ' 我可以将带有名称的字符串拆分为名称列表

    names = re.split(',[ ]*and |,[ ]*| and ', authors)

使用普通的split(" ") 我可以获得比全名更有用的姓氏（姓氏）

    names = [(name, name.split(' ')[-1]) for name in names]

即。

names: [('Christer Samuelsson', 'Samuelsson'), ('Pasi Tapanainen', 'Tapanainen'), ('Atro Voutilainen', 'Voutilainen')]

现在我可以使用这些名字（或者更确切地说是姓氏）和年份来生成像surname (year)、surname, year 这样的字符串，然后在文档中搜索。

如果有很多姓氏，那么我可以获取第一个姓氏并生成surname et al. (year)等。

使用这些字符串和starndard字符串函数text.count(generated_string)我可以计算它们。

目前已经是我的全部了，但还是不理想。

您可以手动找到文档中的所有引用并使用它们来测试代码。您会看到哪些计算正确，哪些需要进行更多更改。

例如，在文本We are using Atro Voutilainen's (1995) 中引用了's。也许应该像在NLP（自然语言处理）中一样使用nltk清理文档

而且一些本地字符会产生问题 - 名称 Järvinen 在一处被提取为 J~rvinen 而在另一处提取为 J/irvinen

import PyPDF2
from PyPDF2.pdf import *  # to import function used in origimal `extractText`

# --- functions ---

def myExtractText(self, distance=None):
    # original code from `page.extractText()`
    # https://github.com/mstamy2/PyPDF2/blob/d7b8d3e0f471530267827511cdffaa2ab48bc1ad/PyPDF2/pdf.py#L2645
    
    text = u_("")

    content = self["/Contents"].getObject()

    if not isinstance(content, ContentStream):
        content = ContentStream(content, self.pdf)
    
    prev_x = 0
    prev_y = 0
    
    for operands, operator in content.operations:
        # used only for test to see values in variables
        #print('>>>', operator, operands)

        if operator == b_("Tj"):
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += _text
        elif operator == b_("T*"):
            text += "\n"
        elif operator == b_("'"):
            text += "\n"
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += operands[0]
        elif operator == b_('"'):
            _text = operands[2]
            if isinstance(_text, TextStringObject):
                text += "\n"
                text += _text
        elif operator == b_("TJ"):
            for i in operands[0]:
                if isinstance(i, TextStringObject):
                    text += i
            text += "\n"
            
        if operator == b_("Tm"):
        
            if distance is True: 
                text += '\n'
                
            elif isinstance(distance, int):
                x = operands[-2]
                y = operands[-1]

                diff_x = prev_x - x
                diff_y = prev_y - y

                #print('>>>', diff_x, diff_y - y)
                #text += f'| {diff_x}, {diff_y - y} |'
                
                if diff_y > distance or diff_y < 0:  # (bigger margin) or (move to top in next column)
                    text += '\n'
                    #text += '\n' # to add empty line between elements
                    
                prev_x = x
                prev_y = y
            
    return text
        
# --- main ---
        
pdfFileObj = open('A97-1011.pdf', 'rb')
pdfReader = PyPDF2.PdfFileReader(pdfFileObj)

text = ''

for page in pdfReader.pages:
    #text += page.extractText()  # original function
    #text += myExtractText(page)        # modified function (works like original version)
    #text += myExtractText(page, True)  # modified function (add `\n` after every `Tm`)
    text += myExtractText(page, 17)  # modified function (add `\n` only if distance is bigger then `17`)   

# get only text after word `References`
pos = text.lower().find('references')

# only referencers as text
references = text[pos+len('references '):]

# doc without references
doc = text[:pos]

# referencers as list
references = references.split('\n')

# remove empty lines and lines which have 2 chars (ie. page number)
references = [item.strip() for item in references if len(item.strip()) > 2]

print('\n--- names ---\n')

data = []

for nubmer, line in enumerate(references, 1): # skip last element with page number
    line = line.strip()
    if line:  # skip empty line
    
        authors_and_year = re.match('((.*)\. (\d{4})\.)', line)
        text, authors, year = authors_and_year.groups()
        #print(text, '|', authors, '|', year)
        
        names = re.split(',[ ]*and |,[ ]*| and ', authors)
        #print(names)
        
        # [(name, last_name), ...]
        names = [(name, name.split(' ')[-1]) for name in names]
        #print(names)
        
        #print(' line:', line)
        print('   text:', text)
        print('authors:', authors)
        print('   year:', year)
        print('  names:', names)
        print('---')
        data.append((authors, names, year))

print('\n--- counting ---\n')

# https://guides.lib.monash.edu/citing-referencing/APA-In-text
# Tapanainen and J/~rvine, 

for authors, names, year in data:
    print('authors:', authors)
    print('   year:', year)
    print('  names:', names)
    print(' et al.:', len(names) > 1)
    print('   and :', len(names) == 2)
    print('---')
    first_lastname = names[0][-1]
    print(doc.count(first_lastname), first_lastname)
    print(doc.count(first_lastname + ', ' + year), first_lastname + ', ' + year)
    print(doc.count(first_lastname + ' (' + year + ')'), first_lastname + ' (' + year + ')')
    
    if len(names) > 1:
        first_lastname_et_al = first_lastname + ' et al.'
        print(doc.count(first_lastname_et_al), first_lastname_et_al)
        print(doc.count(first_lastname_et_al + ', ' + year), first_lastname_et_al + ', ' + year)
        print(doc.count(first_lastname_et_al + ' (' + year + ')'), first_lastname_et_al + ' (' + year + ')')

    if len(names) == 2:
        all_lastnames = ' and '.join(item[-1] for item in names)
        print(doc.count(all_lastnames), all_lastnames)
        print(doc.count(all_lastnames + ', ' + year), all_lastnames + ', ' + year)
        print(doc.count(all_lastnames + ' (' + year + ')'), all_lastnames + ' (' + year + ')')

    print('----------')

名称提取结果：

--- names ---

   text: Arto Anttila. 1995.
authors: Arto Anttila
   year: 1995
  names: [('Arto Anttila', 'Anttila')]
---
   text: Dekang Lin. 1996.
authors: Dekang Lin
   year: 1996
  names: [('Dekang Lin', 'Lin')]
---
   text: Jason M. Eisner. 1996.
authors: Jason M. Eisner
   year: 1996
  names: [('Jason M. Eisner', 'Eisner')]
---
   text: David G. Hays. 1964.
authors: David G. Hays
   year: 1964
  names: [('David G. Hays', 'Hays')]
---
   text: Hans Jiirgen Heringer. 1993.
authors: Hans Jiirgen Heringer
   year: 1993
  names: [('Hans Jiirgen Heringer', 'Heringer')]
---
   text: Richard Hudson. 1991.
authors: Richard Hudson
   year: 1991
  names: [('Richard Hudson', 'Hudson')]
---
   text: Arvi Hurskainen. 1996.
authors: Arvi Hurskainen
   year: 1996
  names: [('Arvi Hurskainen', 'Hurskainen')]
---
   text: Time J~rvinen. 1994.
authors: Time J~rvinen
   year: 1994
  names: [('Time J~rvinen', 'J~rvinen')]
---
   text: Fred Karlsson, Atro Voutilainen, Juha Heikkil~, and Arto Anttila, editors. 1995.
authors: Fred Karlsson, Atro Voutilainen, Juha Heikkil~, and Arto Anttila, editors
   year: 1995
  names: [('Fred Karlsson', 'Karlsson'), ('Atro Voutilainen', 'Voutilainen'), ('Juha Heikkil~', 'Heikkil~'), ('Arto Anttila', 'Anttila'), ('editors', 'editors')]
---
   text: Fred Karlsson. 1990.
authors: Fred Karlsson
   year: 1990
  names: [('Fred Karlsson', 'Karlsson')]
---
   text: Michael McCord. 1990.
authors: Michael McCord
   year: 1990
  names: [('Michael McCord', 'McCord')]
---
   text: Igor A. Mel'~uk. 1987.
authors: Igor A. Mel'~uk
   year: 1987
  names: [("Igor A. Mel'~uk", "Mel'~uk")]
---
   text: Christer Samuelsson, Pasi Tapanainen, and Atro Voutilainen. 1996.
authors: Christer Samuelsson, Pasi Tapanainen, and Atro Voutilainen
   year: 1996
  names: [('Christer Samuelsson', 'Samuelsson'), ('Pasi Tapanainen', 'Tapanainen'), ('Atro Voutilainen', 'Voutilainen')]
---
   text: Daniel Sleator and Davy Temperley. 1991.
authors: Daniel Sleator and Davy Temperley
   year: 1991
  names: [('Daniel Sleator', 'Sleator'), ('Davy Temperley', 'Temperley')]
---
   text: Pasi Tapanainen and Time J/irvinen. 1994.
authors: Pasi Tapanainen and Time J/irvinen
   year: 1994
  names: [('Pasi Tapanainen', 'Tapanainen'), ('Time J/irvinen', 'J/irvinen')]
---
   text: Pasi Tapanainen. 1996.
authors: Pasi Tapanainen
   year: 1996
  names: [('Pasi Tapanainen', 'Tapanainen')]
---
   text: Lucien TesniSre. 1959.
authors: Lucien TesniSre
   year: 1959
  names: [('Lucien TesniSre', 'TesniSre')]
---
   text: Atro Voutilainen. 1995.
authors: Atro Voutilainen
   year: 1995
  names: [('Atro Voutilainen', 'Voutilainen')]
---

计数结果：

--- counting ---

authors: Arto Anttila
   year: 1995
  names: [('Arto Anttila', 'Anttila')]
 et al.: False
   and : False
---
1 Anttila
0 Anttila, 1995
1 Anttila (1995)
----------
authors: Dekang Lin
   year: 1996
  names: [('Dekang Lin', 'Lin')]
 et al.: False
   and : False
---
4 Lin
0 Lin, 1996
1 Lin (1996)
----------
authors: Jason M. Eisner
   year: 1996
  names: [('Jason M. Eisner', 'Eisner')]
 et al.: False
   and : False
---
2 Eisner
2 Eisner, 1996
0 Eisner (1996)
----------
authors: David G. Hays
   year: 1964
  names: [('David G. Hays', 'Hays')]
 et al.: False
   and : False
---
1 Hays
1 Hays, 1964
0 Hays (1964)
----------
authors: Hans Jiirgen Heringer
   year: 1993
  names: [('Hans Jiirgen Heringer', 'Heringer')]
 et al.: False
   and : False
---
1 Heringer
1 Heringer, 1993
0 Heringer (1993)
----------
authors: Richard Hudson
   year: 1991
  names: [('Richard Hudson', 'Hudson')]
 et al.: False
   and : False
---
2 Hudson
1 Hudson, 1991
1 Hudson (1991)
----------
authors: Arvi Hurskainen
   year: 1996
  names: [('Arvi Hurskainen', 'Hurskainen')]
 et al.: False
   and : False
---
1 Hurskainen
0 Hurskainen, 1996
1 Hurskainen (1996)
----------
authors: Time J~rvinen
   year: 1994
  names: [('Time J~rvinen', 'J~rvinen')]
 et al.: False
   and : False
---
2 J~rvinen
1 J~rvinen, 1994
1 J~rvinen (1994)
----------
authors: Fred Karlsson, Atro Voutilainen, Juha Heikkil~, and Arto Anttila, editors
   year: 1995
  names: [('Fred Karlsson', 'Karlsson'), ('Atro Voutilainen', 'Voutilainen'), ('Juha Heikkil~', 'Heikkil~'), ('Arto Anttila', 'Anttila'), ('editors', 'editors')]
 et al.: True
   and : False
---
3 Karlsson
0 Karlsson, 1995
0 Karlsson (1995)
2 Karlsson et al.
1 Karlsson et al., 1995
1 Karlsson et al. (1995)
----------
authors: Fred Karlsson
   year: 1990
  names: [('Fred Karlsson', 'Karlsson')]
 et al.: False
   and : False
---
3 Karlsson
0 Karlsson, 1990
1 Karlsson (1990)
----------
authors: Michael McCord
   year: 1990
  names: [('Michael McCord', 'McCord')]
 et al.: False
   and : False
---
1 McCord
1 McCord, 1990
0 McCord (1990)
----------
authors: Igor A. Mel'~uk
   year: 1987
  names: [("Igor A. Mel'~uk", "Mel'~uk")]
 et al.: False
   and : False
---
1 Mel'~uk
0 Mel'~uk, 1987
1 Mel'~uk (1987)
----------
authors: Christer Samuelsson, Pasi Tapanainen, and Atro Voutilainen
   year: 1996
  names: [('Christer Samuelsson', 'Samuelsson'), ('Pasi Tapanainen', 'Tapanainen'), ('Atro Voutilainen', 'Voutilainen')]
 et al.: True
   and : False
---
1 Samuelsson
0 Samuelsson, 1996
0 Samuelsson (1996)
1 Samuelsson et al.
0 Samuelsson et al., 1996
1 Samuelsson et al. (1996)
----------
authors: Daniel Sleator and Davy Temperley
   year: 1991
  names: [('Daniel Sleator', 'Sleator'), ('Davy Temperley', 'Temperley')]
 et al.: True
   and : True
---
1 Sleator
0 Sleator, 1991
0 Sleator (1991)
0 Sleator et al.
0 Sleator et al., 1991
0 Sleator et al. (1991)
0 Sleator and Temperley
0 Sleator and Temperley, 1991
0 Sleator and Temperley (1991)
----------
authors: Pasi Tapanainen and Time J/irvinen
   year: 1994
  names: [('Pasi Tapanainen', 'Tapanainen'), ('Time J/irvinen', 'J/irvinen')]
 et al.: True
   and : True
---
6 Tapanainen
0 Tapanainen, 1994
0 Tapanainen (1994)
0 Tapanainen et al.
0 Tapanainen et al., 1994
0 Tapanainen et al. (1994)
0 Tapanainen and J/irvinen
0 Tapanainen and J/irvinen, 1994
0 Tapanainen and J/irvinen (1994)
----------
authors: Pasi Tapanainen
   year: 1996
  names: [('Pasi Tapanainen', 'Tapanainen')]
 et al.: False
   and : False
---
6 Tapanainen
3 Tapanainen, 1996
0 Tapanainen (1996)
----------
authors: Lucien TesniSre
   year: 1959
  names: [('Lucien TesniSre', 'TesniSre')]
 et al.: False
   and : False
---
0 TesniSre
0 TesniSre, 1959
0 TesniSre (1959)
----------
authors: Atro Voutilainen
   year: 1995
  names: [('Atro Voutilainen', 'Voutilainen')]
 et al.: False
   and : False
---
3 Voutilainen
0 Voutilainen, 1995
1 Voutilainen (1995)
----------