Document Type

Poster

Organization

Southwestern Oklahoma State University

Conference Title

SWOSU Research Fair

City and State

Weatherford, Oklahoma

Conference Date

Nov 21, 2019

Publication Date

11-21-2019

Abstract

Over the summer, I was given access to WebScraper3000 and asked to web scrape, collect internet data, from ratemyprofessors.com. I spent several months working with the program to accomplish this. It only ever seemed to work properly on websites where it has tutorials about scraping them. Even then, the program would only scrape the most rudimentary data in small amounts. If either of these two conditions were missing, the software would lose large quantities of data or just simply crash along the way. After collecting enough data to act as a sample for the research, I decided to abandon the software and begin looking into building my own alternative. I decided on using the Python programming language and its beautifulsoup4 import. The result ended with a far smoother and more complete data collection and set.

Share

COinS