django-dynamic-scraper - مستندات

Django Dynamic Scraper (DDS) یک برنامه برای ساخت جنگو در بالای چارچوب Scrapy است. در حالی که بسیاری از ویژگی های Scrapy را حفظ می کند، به شما امکان می دهد که از طریق رابط کاربری Django به صورت پویا عنکبوت را ایجاد و مدیریت کنید.

Note

Lot’s of new features added recently :

• Django 1.10/1.11/Scrapy 1.4 support
• Python 3.4+ support
• Javascript rendering
• Scraping JSON content
• More flexible ID and detail page URL(s) concept
• Several checkers for a single scraper
• Custom HTTP Header/Body, Cookies, GET/POST requests
• Scrapy Meta attributes
• Scraper/Checker Monitoring

See Release Notes for further details!

Features

• Create and manage scrapers for your Django models in the Django admin interface
• Many features of Scrapy like regular expressions, processors, pipelines (see Scrapy Docs)
• Image/screenshot scraping
• Dynamic scheduling depending on crawling success via Django Celery
• Checkers to check if items once scraped are still existing

User Manual

• Introduction
• Installation
  • Requirements
  • Release Compatibility Table
  • Installation with Pip
  • Manual Installation
  • Setting up Scrapy
• Getting started
  • Creating your Django models
  • Defining the object to be scraped
  • Defining your scrapers
  • Connecting Scrapy with your Django objects
  • Running/Testing your scraper
• Advanced topics
  • Defining/Running item checkers
  • Scheduling scrapers/checkers
  • Advanced Request Options
  • Pagination
  • Scraping JSON content
  • Scraping images/screenshots
  • Where to go from here
• Basic services
  • Logging / Log Markers
  • Monitoring
• Reference
  • Settings
  • Django Model Reference
  • API Reference
  • Processors
• Development
  • How to contribute
  • Running the test suite
  • Release Notes
  • Roadmap

Indices and tables

• Index
• Module Index
• Search Page