Skip to content

custom-http-requests.mdx

Latest commit

 

History

History
286 lines (236 loc) · 6.94 KB

custom-http-requests.mdx

File metadata and controls

286 lines (236 loc) · 6.94 KB
title Custom HTTP Requests
description Learn how to customize HTTP requests with custom headers, authentication, and API interactions for html2rss.

import { Code } from "@astrojs/starlight/components";

Some sites only work when requests carry the headers, tokens, or cookies your browser uses. html2rss supports those cases without changing the rest of your feed workflow.

Keep this structure in mind:

  • headers stays top-level
  • strategy stays top-level
  • request-specific controls such as budgets and strategy-specific options live under request

When You Need Custom Headers

You might need custom HTTP requests when:

  • APIs require authentication (Bearer tokens, API keys)
  • Websites block default user agents (need to appear as a real browser)
  • Content is behind login (session cookies, authorization headers)
  • Rate limiting (custom headers to identify your requests)
  • Content negotiation (specific Accept headers for different formats)

Basic Configuration

Add a headers section to your feed configuration. This example is a complete, valid config:

<Code code={headers: User-Agent: "Mozilla/5.0 (compatible; html2rss/1.0)" Authorization: "Bearer YOUR_API_TOKEN" Accept: "application/json" channel: url: https://api.example.com/posts selectors: items: selector: "array > object" title: selector: "title" url: selector: "url"} lang="yaml" />

Request Controls

Request budgets are configured under request, not as top-level keys:

<Code code={headers: User-Agent: "Mozilla/5.0 (compatible; html2rss/1.0)" request: max_redirects: 5 max_requests: 6 channel: url: https://example.com/articles selectors: items: selector: article title: selector: h2 url: selector: a extractor: href} lang="yaml" />

  • request.max_redirects limits redirect hops
  • request.max_requests limits the total request budget for the feed build
  • request.browserless.* is reserved for Browserless-only behavior such as preload actions
  • request.botasaurus.* is reserved for Botasaurus-only behavior such as navigation mode and retries

Common Use Cases

API Authentication

Many APIs require authentication tokens:

<Code code={headers: Authorization: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." X-API-Key: "your-api-key-here" channel: url: "https://api.example.com/posts" selectors: items: selector: "array > object" title: selector: "title" url: selector: "url"} lang="yaml" />

User Agent Spoofing

Some websites block requests that don't look like real browsers:

<Code code={headers: User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" Accept-Language: "en-US,en;q=0.5" Accept-Encoding: "gzip, deflate" channel: url: "https://example.com/articles" selectors: items: selector: "article" title: selector: "h2" url: selector: "a" extractor: "href"} lang="yaml" />

Content Type Negotiation

Request specific content types:

<Code code={headers: Accept: "application/json" channel: url: "https://api.example.com/posts" selectors: items: selector: "array > object" title: selector: "title" url: selector: "url"} lang="yaml" />

Custom API Headers

Some APIs require specific headers:

<Code code={headers: X-Requested-With: "XMLHttpRequest" X-Custom-Header: "your-value" Content-Type: "application/json" channel: url: "https://api.example.com/posts" selectors: items: selector: "array > object" title: selector: "title" url: selector: "url"} lang="yaml" />

Dynamic Headers

You can use dynamic parameters in headers for runtime values:

<Code code={headers: Authorization: "Bearer %<api_token>s" X-User-ID: "%<user_id>s" channel: url: "https://api.example.com/users/%<user_id>s/posts" selectors: items: selector: "array > object" title: selector: "title" url: selector: "url"} lang="yaml" />

See our Dynamic Parameters guide for more details.

Notes

  • Header examples that target third-party APIs are illustrative. Authentication requirements, header names, and response shapes can change independently of html2rss.
  • For JSON APIs, validate the response structure before assuming selectors like array > object or html_url will match.
  • If you document or share a config for reuse, prefer placeholder values and parameterized headers over embedding real tokens.

Testing Your Headers

Test your configuration to ensure headers work correctly:

<Code code={curl -H "Authorization: Bearer YOUR_TOKEN" \ https://api.example.com/posts && \ html2rss feed your-config.yml} lang="bash" />

Troubleshooting

Common Issues

  • 401 Unauthorized: Check your authentication headers
  • 403 Forbidden: Verify API keys and permissions
  • 429 Too Many Requests: Add rate limiting or different user agents
  • Empty responses: Some APIs require specific Accept headers

Debug Tips

  1. Use browser developer tools to see what headers successful requests use
  2. Test with curl before configuring html2rss
  3. Check API documentation for required headers
  4. Enable debug logging to see what headers are being sent

Advanced Examples

GitHub API

<Code code={headers: Authorization: "token YOUR_GITHUB_TOKEN" Accept: "application/vnd.github.v3+json" User-Agent: "html2rss/1.0" channel: url: https://api.github.com/repos/owner/repo/issues selectors: items: selector: "array > object" title: selector: "title" url: selector: "html_url"} lang="yaml" />

Reddit API

<Code code={headers: User-Agent: "html2rss/1.0 by your-username" Accept: "application/json" channel: url: https://www.reddit.com/r/programming.json selectors: items: selector: "data > children > object > data" title: selector: "title" url: selector: "url"} lang="yaml" />

Related Topics

Need More Help?