Launching Crawlio

Hey everyone, I’m excited to share Crawlio, a web scraping API I helped build to make scraping simpler, more accessible, and hopefully a little less frustrating for developers like us.

Why Crawlio?

If you’ve done any scraping, you know how tricky it can be to juggle things like robots.txt rules, rate limits, and scraping at scale. Crawlio tries to handle those complexities under the hood so you can focus on what matters — your data and your app.

It offers three main endpoints:

• /scrape — for grabbing data from a single page
• /crawl — to recursively crawl entire sites
• /batch-scrape — to scrape multiple URLs in one request

It’s still in public beta, so there might be bugs, but that also means your feedback directly shapes the tool — plus you earn free scrape credits for valid bug reports.

How Crawlio Compares to Other Tools

You might have heard of tools like FireCrawl or Crawl4AI, so here’s a quick, honest comparison from my perspective:

• Pricing: Crawlio is generally cheaper than FireCrawl, which can be a big deal if you’re on a budget or just experimenting.
• Features: Right now, Crawlio doesn’t have some of the advanced features those platforms offer — think of things like super fine-grained filters or certain integrations. But we’re actively developing, and new features roll out every week. So while we might not cover everything just yet, we’re catching up fast.
• Approach: We take responsible scraping seriously — respecting robots.txt out of the box. That’s non-negotiable for us.

The idea is to offer a solid, reliable base that’s easy to get started with, then grow it with input from developers using it day-to-day.

Getting Started

If you want to give it a try, here’s a simple example in Python:

from crawlio.client import Crawlio

client = Crawlio(api_key="your-api-key")

result = client.scrape({
    "url": "https://example.com",
    "markdown": True
})

print(result["markdown"])

Or Node.js:

import Crawlio from "crawlio-js";

const client = new Crawlio({ apiKey: "your-api-key" });

const result = await client.scrape({ url: "https://example.com" });
console.log(result.html);

Grab your API key by signing up at Crawlio.

Why Public Beta?

Launching in public beta means things aren’t perfect yet — bugs will happen, and features will roll out over time. But it also means your feedback has a direct impact, and you get rewarded with free scrape credits for valid bug reports. It’s a way for us to build something that actually works for developers like you and me.

Final Thoughts

If you want a scraping API that’s affordable, respects site rules, and is evolving fast, Crawlio might be worth a look. I’m excited to keep building it and hearing what you think.

Thanks for checking it out — and happy scraping! - kisshan13