The simplest way to debug a Scrapy spider in PyCharm

  • Add the following code to your Scrapy spider(PROJECT_FOLDER/PROJECT_NAME/spiders/your_spider.py), or create a new script and paste the following code into it(PROJECT_FOLDER/start.py).
if __name__ == '__main__':    from scrapy.crawler import CrawlerProcess    from scrapy.utils.project import get_project_settings    process = CrawlerProcess(get_project_settings())    process.crawl('your_spider_name')    process.start()
  • In PyCharm, right click your spider script(your_spider.py) or the script to start your spider(start.py), then select “Debug,” and your are good to go.

--

--

--

A Python web scraping expert.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

New concept of travel planing on web

AN UPDATE FROM SIMON BENSON, CTO.

CodeSubmit for Mobile App Developer Assessment

LoRa LPWAN — Long Range Low Power Wide Area Network

Photo by Josh Hild from Pexels

How to connect a Google Assistant Action to a custom chatbot backend

Google Assistant family: Google Home and Nest Hub

Angular, SEO, and pre-rendering with Netlify

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nathan Chen

Nathan Chen

A Python web scraping expert.

More from Medium

Fizzbuzz in 2022

Build Your First Pypi Package

Misconceptions about Parser

Ansi-color.cmd: Is Windows Batch Scripting a Programming Language?

A grid of foreground and background colors showing the resulting combinations