How to get scraped

300 · Onur Solmaz · July 15, 2026, 7:48 a.m.
Summary
This blog post discusses strategies for optimizing a website to be easily scraped by AI crawlers, arguing that becoming 'agent-famous' is crucial for modern careers. It details technical steps such as serving content as static HTML, adjusting robots.txt to welcome crawlers, and using a permissive license to enhance discoverability. The author emphasizes the importance of writing valuable content that is also legible to machines, asserting that successful publishing in an AI-driven world requires thoughtful technical optimization alongside meaningful writing.