فلش‌کارت‌های آموزشی مهندسی داده: جمع‌آوری داده از وب با Scrapy و Beautiful Soup

🎓 دوره آموزشی جامع

📚 اطلاعات دوره

عنوان دوره: دوره مهندسی داده: جمع‌آوری داده از وب با Scrapy و Beautiful Soup

موضوع کلی: برنامه نویسی

موضوع میانی: مهندسی داده (Data Engineering)

📋 سرفصل‌های دوره

• 1. Course Overview: Web Data Collection in Data Engineering Context
• 2. Python Environment Setup for Scrapy and Beautiful Soup
• 3. HTML Fundamentals for Effective Parsing Strategies
• 4. Initializing Your First Scrapy Project Structure
• 5. Building Basic Scrapy Spiders for Static Pages
• 6. Defining and Using Scrapy Items for Data Modeling
• 7. Implementing Scrapy Item Pipelines for Processing
• 8. Mastering CSS Selectors in Scrapy Extraction
• 9. Advanced XPath Techniques for Precise Node Targeting
• 10. Integrating Beautiful Soup for Supplementary Parsing
• 11. Beautiful Soup Installation and Import Configuration
• 12. Using Find and Find_all Methods for Data Extraction
• 13. Navigating Parse Trees with Beautiful Soup Methods
• 14. Extracting Text Content and HTML Attributes
• 15. Data Cleaning and Normalization Post-Extraction
• 16. Exporting Scraped Data to JSON Format
• 17. Exporting Scraped Data to CSV Format
• 18. Storing Data in Relational Databases via Pipelines
• 19. Storing Data in NoSQL Document Stores
• 20. Customizing Scrapy Feed Exports for Specific Formats
• 21. Configuring Request Headers for Anonymity
• 22. Managing Proxies and IP Rotation in Scrapy
• 23. Handling Cookies and Session Persistence
• 24. Automating Pagination Logic in Spiders
• 25. Implementing Recursive Crawling for Multi-Level Sites
• 26. Link Extraction and Follow Strategies
• 27. Error Handling and Retry Policies in Scrapy
• 28. Setting Up Comprehensive Logging Systems
• 29. Using Scrapy Shell for Real-Time Debugging
• 30. Customizing User-Agents and Request Fingerprinting
• 31. Compliance with Robots.txt and Ethical Guidelines
• 32. Legal Frameworks for Web Data Collection
• 33. Integrating Scraped Data with REST APIs
• 34. Data Transformation Enrichment Within Pipelines
• 35. Validating Data Integrity and Schema Compliance
• 36. Scheduling Spiders with Cron and Systemd
• 37. Containerizing Scrapy Applications Using Docker
• 38. Introduction to Asynchronous Scraping with AIOHTTP
• 39. Handling HTML Forms and POST Requests
• 40. Rendering JavaScript Content with Playwright Integration
• 41. Best Practices for Large-Scale Scraping Operations
• 42. Performance Optimization Techniques for Crawlers
• 43. Memory Management During Long-Running Jobs
• 44. Unit Testing Scrapy Spiders Effectively
• 45. CI/CD Integration for Data Engineering Pipelines
• 46. Version Control Strategies for Scraping Projects
• 47. Documenting Scraped Data Workflows and Outputs
• 48. Data Quality Assurance and Validation Protocols
• 49. Implementing Rate Limit and Throttling Strategies
• 50. Capstone Project: End-to-End Data Collection Pipeline