如何找到该网站的 xpath 表达式？

Question

我想抓取这个网页。

我有很多简历，我的任务是收集每一份的技能。 这是网页的链接 ==> https://www.livecareer.com/resume-search/search?jt=software%20engineer

Answer 1

这里其实不用selenium。您可以使用BeautifulSoup 轻松完成此操作。完整代码如下：

import requests
from bs4 import BeautifulSoup

r = requests.get('https://www.livecareer.com/resume-search/search?jt=software%20engineer').text

soup = BeautifulSoup(r,'html.parser')

ul = soup.find('ul',class_ = 'resume-list list-unstyled')

li_items = ul.find_all('li')[1:]

links = []

for li in li_items:
    links.append('https://www.livecareer.com/'+li.a['href'])

skills = []

for link in links:
    
    r = requests.get(link).text
    soup = BeautifulSoup(r,'html.parser')
    div = soup.find('div',class_ = 'field singlecolumn')
    skills.append(div.text)

print(skills)

输出：

['agile, AutoCAD, C++, CAD, Oral, data entry, database, Engineer in Training, EIT, Engineering analysis, XML, functional, GUI, HTML, JavaScript, Team leadership, Lockheed Martin, macros, Manufacturing processes, MATLAB, mechanical, meetings, Excel, Organizational skills, presentations, Process improvement, program management, programming, Project planning, Python, research, scrum, Six Sigma, Software development, Solidworks, SQL, switches, telemetry, video, Web design, website, Written communication', "Senior Outreach at Senior Center\xa0Planned and organized a joint celebration of the Chinese New Year with the collaboration of the Westborough Public Schools. \xa0Promoted cultural awareness and broke the language barrier of different races of backgrounds.Volunteer at ChurchIdentified problems and implemented a process to eliminate a data-entry camp registration process by 100% by building new online Registration Forms for registration and the student's cultural classes arrangement.Designed posters, flyers and presentation slides with graphics and photos for the different organizational events with Microsoft Word and PowerPointDeveloped a structural documentation on publishing an annual report with detailed steps and instructions on the process that are easy to follow and quickly learn by others. \xa0Implemented a 30th Anniversary Special Edition project in a commercial quality of work with excellent time management skills to meet the deadline.Cayenne SoftwareExperienced team spirit in effort of reducing the workload of bugs fixes on the software product.\u200bAllmerica Financial CompanyProvided a sole support to the Hanover 1099 system with strong commitment and responsibility. Fined tuned the system resulting in cost savings for the Allmerica Financial Company. Winner of the Gold Crown Customer Recognition award.", 'Motivated Software Engineer seeking employment as part of a dynamic software development team. Fluent in C,C++,JAVA and python.', 'Developed peer-to-peer secure file transfer system in JAVA.This involved the application of symmetric\r\n     and asymmetric key cryptography algorithms, and JAVA concepts like multi-threading, socket\r\n     programming, etc.Implemented a system to query XML in JAVA.The query language was a subset of XPath\r\n    Modeled a project "Personal Health Management System" using UML and implemented it in Visual C#.The code was tested using NUnit.Object oriented software development process was used for this\r\n     project\r\n    Developed a \'license plate game\' in C on LINUX o/s using client/server architecture.This required the\r\n     application of distributed programming concepts like Sockets, RPC, multi-threading, etc.RESEARCH PAPER:\r\n    XMorph: A Shape-Polymorphic, Domain-Specific XML Data Transformation Language,\r\n     International Conference on Data Engineering (ICDE 2010), IEEE CS, Los Angeles, USA, March 2010.', 'Performance evaluation of In-Kernel System Call Implemented and evaluated In-kernel system call using dynamic loadable kernel module on x_86_64 architecture.Re-Development free approach to migrate Java applications to cloud at College of Engineering, Pune Implemented file access sub-system of a WebJDK which leverages the File-System API provided by HTML5.This allows the use of standard Java APIs for accessing client files.2016 2013.', 'Accomplished Computer Technician with a rapidly increasing range of industry experience looking to bring strong instincts and a proven record of procedural compliance, process management and strong operational skills to a rapidly growing company. ', 'Seeking a fulltime position as a Developer / Systems Admin / DBA for a company needing a hard working, \r\ntaskoriented person with an indepth understanding of software development and database tuning.', '3 Years of experience in Information Technology with emphasis on Design, Development and End to End Implementation of Consulting based solutions with expertise on working with Object Orient Analysis and Design using Java/J2EE Technologies viz. JSP/Servlets/EJB,JDBC, Web services , Web sockets, Spring Frameworks, Spring-boot, Angular, JQuery, XML/XSLT, JSON, Integration Developer Service Component Architecture & Service Data Objects, Rational Application Developer, Test Driven Development using JUnit, Jenkins, GIT, Cloud Foundry,  Eclipse/Intelij IDE, UNIX, Gradle Scripts, DB2/Oracle/MySQL Databases.', '.NET 3.5, .NET, ASP .NET 3.5, ASP.NET 2.0, ASP.NET 3.5, AJAX, ASM, Banking, Basic, Business Objects, c, CSS, CSS 2, customer satisfaction, data analysis, Database, delivery, EBusiness, editor, Electronics, HP, HTML 4, HTML, IDE, IIS 7.0, ITIL, JavaScript, C#, C# 3.0, Windows, windows applications, 2000, 3.1, Windows 98, Enterprise, Oct, Operating systems, Oracle 9, Oracle database, PL/SQL, personnel, programming, recording, reporting, sales, Servers, Service Level Agreement, SLA, Visual SourceSafe, Visual  SourceSafe, SQL, SQL Server, technical support, TOAD, UNIX, vi, Microsoft Visual Studio, Visual studio, Windows server', 'Represent Stanford  Ballroom Dance team in various competitions in the Bay area.\r\n*Represented University of Maryland in Ballroom dance competitions in UMD, UPenn, MIT, Columbia University & Ohio \r\n*Have a keen interest in photography, especially of dancers in motion.']

你也可以通过在你的代码中添加这些行来创建一个漂亮的DataFrame（为了更好的可读性）：

dictionary = {'Links':links,
              'Skills':skills}

df = pd.DataFrame(dictionary)

print(df)

输出：

                                                                                                           
                Skills                                                   Links
0  https://www.livecareer.com//resume-search/r/so...  agile, AutoCAD, C++, CAD, Oral, data entry, da...
1  https://www.livecareer.com//resume-search/r/so...  Senior Outreach at Senior Center Planned and o...
2  https://www.livecareer.com//resume-search/r/so...  Motivated Software Engineer seeking employment...
3  https://www.livecareer.com//resume-search/r/so...  Developed peer-to-peer secure file transfer sy...
4  https://www.livecareer.com//resume-search/r/so...  Performance evaluation of In-Kernel System Cal...
5  https://www.livecareer.com//resume-search/r/so...  Accomplished Computer Technician with a rapidl...
6  https://www.livecareer.com//resume-search/r/so...  Seeking a fulltime position as a Developer / S...
7  https://www.livecareer.com//resume-search/r/so...  3 Years of experience in Information Technolog...
8  https://www.livecareer.com//resume-search/r/so...  .NET 3.5, .NET, ASP .NET 3.5, ASP.NET 2.0, ASP...
9  https://www.livecareer.com//resume-search/r/so...  Represent Stanford  Ballroom Dance team in var...

希望这会有所帮助！

Answer 2

要把这个扔出去.....

如前所述，使用 chrome 右键单击您想要的元素 点击检查 按 ctrl + f 调出搜索窗口

然后编写您的 xpath，使其返回一个（且只有一个）页面对象。

例如：

//a[contains(text(), 'my text')] 
//div[@id='myDivID']

为项目创建 xpath。永远不要使用“复制 Xpath”选项。您将获得如下路径，这是您可以编写的最脆弱的 xpath。

//*[@id="wrapper"]/div[2]/div[2]/div[1]/aside[1]/div/div/div[2]/div/div[1]/a

如果你不熟悉编写 xpath，请到这里阅读https://www.w3schools.com/xml/xpath_intro.asp

您在这里遇到的一个问题是文本在 div 标记内，并且确实应该在跨度内。您可以尝试以下方法：

//div[@class='field singlecolumn']/text()

Answer 3

您可以使用浏览器的开发工具来查找任何 html 元素的 XPath。以下示例适用于 Chrome，但其他浏览器的步骤将非常相似：

1. 右键单击页面上要了解其 XPath 的项目。
2. 单击“检查”。这将打开开发工具，并突出显示相关元素。
3. 如果您所关注的元素不是突出显示的元素，请在显示的交互式 html 中四处浏览。通常，当您悬停某个元素时，匹配项会在主页中突出显示。
4. 右键单击导航器中的 html 元素。
5. 选择“复制 -> 复制 XPath”。

您在抓取这些页面时可能遇到的主要问题是，您的目标可能每次都不在同一个位置。如果这些是用户创建的文档，那么它们每次可能具有不同的布局，因此 XPath 可能与每个页面上的相同部分不匹配。如果是这种情况，您可能需要采用更复杂的方法（jQuery、Selenium、Cypress 等都有用于按文本内容搜索、在父/子元素之间导航等工具）