Skip to content

hoangson0409/Merchant_Website_Crawler

Repository files navigation

Django

Running in outer Django project folder (cd django_project)

  • Make Migrations python manage.py makemigrations
  • Migrate to Database python manage.py migrate
  • Run Server python manage.py runserver

Scrapy & Scrapyd

Run in scrapy application (cd django_project/scrapy_app/scrapy_app)

  • Run Scrapyd Server scrapyd
  • Deploy spiders scrapyd-deploy scrapy_app -p spiderman
  • List spiders scrapyd-client spiders -p spiderman
  • Cancel Job curl http://localhost:6800/cancel.json -d project=spiderman -d job=<job_id>

CELERY

Running in outer Django project folder (cd django_project)

  • Start Worker celery -A django_project worker -P solo
  • Start Beat (Scheduler) celery -A django_project beat

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages