Document Type
Poster
Organization
Southwestern Oklahoma State University
Conference Title
SWOSU Research Fair
City and State
Weatherford, Oklahoma
Conference Date
Nov 21, 2019
Publication Date
11-21-2019
Abstract
Over the summer, I was given access to WebScraper3000 and asked to web scrape, collect internet data, from ratemyprofessors.com. I spent several months working with the program to accomplish this. It only ever seemed to work properly on websites where it has tutorials about scraping them. Even then, the program would only scrape the most rudimentary data in small amounts. If either of these two conditions were missing, the software would lose large quantities of data or just simply crash along the way. After collecting enough data to act as a sample for the research, I decided to abandon the software and begin looking into building my own alternative. I decided on using the Python programming language and its beautifulsoup4 import. The result ended with a far smoother and more complete data collection and set.
Recommended Citation
Clark, Kurtis and Evert, Amanda, "Building an Alternative Web Scraper for Big Data Analytics" (2019). Student Research. 24.
https://dc.swosu.edu/cpgs_edsbt_bcs_student/24