$ python add_site.py --yes https://www.recipetineats.com
Working directory: /app
------------------------------------------------------------
============================================================
Adding support for: recipetineats.com
Using model: deepseek/deepseek-v3.2
============================================================
============================================================
COMPREHENSIVE SITE ANALYSIS
Model: deepseek/deepseek-v3.2
============================================================
Step 0: Gathering site data...
Fetching URLs from homepage...
Found 100 URLs on homepage
Fetching sitemap...
Found sitemap index with 8 sitemaps
Found 2034 URLs in sitemap (290 category URLs)
Analyzing navigation structure...
Found 85 navigation items (71 likely categories)
Step 1: Analyzing URL patterns...
Recipe pattern: ^https://www\.recipetineats\.com/[a-z0-9-]+/(?:#comments)?$
URL structure: flat
Confidence: high
Step 2: Analyzing category structure...
ERROR in Step 2:
Step 3: Validating configuration...
Pattern valid: False
Unwanted valid: False
Categories valid: True
Issues: ["Recipe pattern incorrectly matches 'https://www.recipetineats.com/categories/' (a non-recipe URL) because the pattern allows any path segment after the domain as long as it contains lowercase letters, numbers, and hyphens, which matches 'categories'.", "Unwanted patterns list includes '/categories/' but the pattern '/category/' is also present; however, '/categories/' is not explicitly listed in unwanted patterns, leading to the non-recipe URL 'https://www.recipetineats.com/categories/' not being filtered.", "Unwanted patterns list includes '/policy-' which should filter 'https://www.recipetineats.com/policy-use-of-recipes-images/', but this URL is listed in the test as a recipe URL (should match), creating a conflict. Based on the URL structure, it appears non-recipe and should be filtered."]
============================================================
ANALYSIS SUMMARY
============================================================
Recipe pattern: ^https://www\.recipetineats\.com/[a-z0-9-]+/(?:#comments)?$
Unwanted patterns: 11 patterns
Has categories: False
Confidence: high
Validation issues:
⚠️ Recipe pattern incorrectly matches 'https://www.recipetineats.com/categories/' (a non-recipe URL) because the pattern allows any path segment after the domain as long as it contains lowercase letters, numbers, and hyphens, which matches 'categories'.
⚠️ Unwanted patterns list includes '/categories/' but the pattern '/category/' is also present; however, '/categories/' is not explicitly listed in unwanted patterns, leading to the non-recipe URL 'https://www.recipetineats.com/categories/' not being filtered.
⚠️ Unwanted patterns list includes '/policy-' which should filter 'https://www.recipetineats.com/policy-use-of-recipes-images/', but this URL is listed in the test as a recipe URL (should match), creating a conflict. Based on the URL structure, it appears non-recipe and should be filtered.
Suggestions:
💡 Update the recipe pattern to exclude paths that start with common non-recipe segments like 'categories', 'recipes', 'contact', etc., or rely more on unwanted patterns for filtering.
💡 Add '/categories/' to the unwanted patterns list to ensure it is filtered.
💡 Review the test URLs: 'https://www.recipetineats.com/policy-use-of-recipes-images/' should likely NOT match the recipe pattern (it is a policy page), so adjust test expectations or ensure unwanted pattern '/policy-' filters it.
💡 Consider adding '/recipes/' to unwanted patterns to filter 'https://www.recipetineats.com/recipes/'.
💡 Add '/contact/' to unwanted patterns to filter 'https://www.recipetineats.com/contact/'.
============================================================
GENERATED SCRAPER CODE
============================================================
"""Recipetineats scraper.
Generated by add_site.py with comprehensive LLM analysis.
URL structure: flat
Recipe URLs follow a flat structure: domain followed by a hyphenated recipe name slug (e.g., /one-pot-chili-mac-and-cheese/). Non-recipe URLs include category pages (/category/), collections (/collections/), blog index (/blog/), policy pages, travel guides, and other non-recipe content. The pattern excludes URLs with extra path segments beyond the initial slug.
"""
import re
from recipe_urls._abstract import AbstractScraper
class RecipetineatsScraper(AbstractScraper):
RECIPE_PATTERN = re.compile(r"^https://www\.recipetineats\.com/[a-z0-9-]+/(?:#comments)?$")
UNWANTED_PATTERNS = [
re.compile(pattern)
for pattern in ['/category/', '/collections/', '/blog/', '/policy-', '/free-recipe-books/', '/new-york-food-map/', '/tokyo-travel-guide-', '/annual-christmas-selfie-bloopers/', '/categories/', '/recipes/', '/contact/']
]
@classmethod
def host(cls):
return "recipetineats.com"
✅ Saved: /app/data/scrapers/recipetineats.py
============================================================
SUCCESS
============================================================
Site added: recipetineats.com
Test it:
python import_site.py https://www.recipetineats.com --dry-run
------------------------------------------------------------
✅ Completed successfully