python - Scrapy put two spiders in single file

Question

I have written two spiders in single file. When I ran scrapy runspider two_spiders.py, only the first Spider was executed. How can I run both of them without splitting the file into two files.

two_spiders.py:

import scrapy

class MySpider1(scrapy.Spider):
    # first spider definition
    ...

class MySpider2(scrapy.Spider):
    # second spider definition
    ...

Answer 1

Let's read the documentation:

Running multiple spiders in the same process

By default, Scrapy runs a single spider per process when you run scrapy crawl. However, Scrapy supports running multiple spiders per process using the internal API.

Here is an example that runs multiple spiders simultaneously:

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

process = CrawlerProcess()
process.crawl(MySpider1)
process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished

(there are few more examples in the documentation)

From your question it is not clear how have you put two spiders into one file. It was not enough to concatenate content of two files with single spiders.

Try to do what is written in the documentation. Or at least show us your code. Without it we can't help you.