Twitter Scraper: Get Data Without the $100/mo API

In February 2023, Twitter's free API tier disappeared overnight. Elon Musk's acquisition killed it — replaced by a $100/month Basic plan capped at 10,000 tweet reads per month. For any research or monitoring use case that involves more than 333 tweets per day, the API is effectively unusable. Most teams quietly stopped using it.

Here's what most guides miss: the API paywall only blocks API access. Public profiles, tweets, and follower counts are still visible in any browser — and a Twitter scraper running inside a real browser session can extract that data without touching the API at all. This guide covers exactly what's extractable, which methods work in 2026, and why most Python approaches fail before the first request completes.

What Happened to the Twitter API — and What It Means for Scrapers

Twitter's free API tier was removed in February 2023. The replacement Basic plan costs $100/month and allows only 10,000 tweet reads per month — roughly 333 tweets per day. The Pro tier costs $5,000/month. Most scraping use cases that previously used the free API now require either the paid API or a browser-based scraping method.

Twitter's API was free with generous rate limits from 2006 through most of 2022. Developers built monitoring tools, research pipelines, academic datasets, and sentiment analysis workflows on top of it. Then in October 2022, the acquisition closed. By February 2023, the free tier was gone.

The practical impact: a researcher tracking a brand keyword that generates 500 mentions per day exhausts the Basic tier's monthly quota in 20 days. The API is priced to push teams toward Pro — at $5,000/month, it's out of reach for most independent researchers, agencies, and small businesses.

The API paywall only blocks API access. Public tweets, profiles, and follower counts are still visible in any browser — and still extractable without touching the API.

This is the key insight most coverage misses. Twitter/X did not make its public data private. Public profiles still show the last 20–40 tweets, follower counts, bio, and engagement data. The platform just removed the clean programmatic interface for accessing that data at scale. Browser-based scraping fills that gap.

What Data Can You Actually Scrape From Twitter/X Without the API?

Without the Twitter API, you can still scrape public profile data (username, bio, follower/following count, joined date), the visible tweet timeline (last 20–40 tweets, text, engagement counts, dates), and hashtag/keyword search results visible in browser. Private accounts, DMs, full historical tweet archives, and algorithmic timeline contents require API access.

Public data on X is defined as content visible to a logged-in or logged-out user without any authentication beyond having an account. For public profiles — which includes virtually all brand, creator, and business accounts — this is substantial:

What's extractable from public profiles

• Username and display name
• Bio text and profile URL
• Location (if set by the user)
• Follower count and following count
• Account creation date (joined date)
• Pinned tweet (if any)
• Last 20–40 tweets visible on the timeline (text, date, likes, retweets, replies)
• Verified status (blue checkmark tier)

What requires the paid API

• Full tweet history beyond what's visible on the timeline
• DMs and private account content
• Filtered streams (real-time tweet monitoring by keyword at scale)
• Analytics data (impressions, link clicks, video views)
• Full follower/following lists beyond what loads in the browser

Why Python requests Fails on Twitter/X Before the First Tweet Loads

Python's requests library gets blocked on Twitter/X with a ~95% failure rate because X requires JavaScript to render tweets and uses a guest_token authentication system that a plain HTTP client cannot replicate. The HTML returned to requests contains no tweet data — just a React shell that requires JavaScript execution to populate.

Twitter/X is a React single-page application. When you visit a profile with requests, you receive an HTML shell — a `

` with no content. The actual tweets load via authenticated GraphQL API calls that happen after JavaScript executes. requests never runs JavaScript, so it never sees a single tweet.

Even if you try to replicate the API calls manually, you hit the guest_token system. Every X session — including logged-out browser visits — receives a short-lived guest_token that authenticates internal GraphQL requests. This token is generated server-side, rotates every 10–15 minutes, and is tied to the browser session. Replicating it in a plain HTTP client requires reverse-engineering the token generation logic, which breaks every time X updates it.

The snscrape failure in 2023

Before 2023, snscrape was the dominant Python library for Twitter scraping — no API key required, 1,000+ GitHub stars, widely used in academic research. In April 2023, X changed its internal API endpoints and authentication flow. snscrape broke. Its GitHub repo has been unmaintained since, with 200+ open issues citing authentication failures. Most Python tutorials that reference snscrape are now giving advice for a tool that no longer works.

snscrape broke in April 2023 when X changed its internal auth flow. 200+ open GitHub issues, no fix. Any tutorial that recommends snscrape is outdated.

Twitter Scraper Python: What Actually Works in 2026 (twscrape)

twscrape is the current best Python library for Twitter/X scraping without the official API. It uses real Twitter account credentials (not API keys) to authenticate requests and access public data. Block rate is approximately 18% with fresh accounts on residential IPs, dropping to ~8% with session rotation. Setup requires creating one or more burner X accounts.

twscrape (GitHub: vladkens/twscrape) is the spiritual successor to snscrape. Instead of trying to replicate X's internal token system, it uses real X account credentials to authenticate — treating itself as a logged-in user rather than an anonymous client. This is the only approach that reliably bypasses the guest_token problem.

Basic setup with twscrape

1. pip install twscrape
2. Create a burner X account (or use an existing one with low activity)
3. Add the account credentials to twscrape's account pool
4. Run searches or profile lookups — twscrape handles authentication automatically
5. Rotate accounts if you hit rate limits (one account = ~150 requests/15-min window)

The limitation: each account is subject to X's rate limits — the same limits that apply to real users. If you need to scrape at high volume, you need multiple accounts. Accounts that exhibit non-human patterns (too many requests too fast, no organic activity) get suspended. twscrape works well for moderate-scale research but is not a replacement for the Pro API at enterprise scale.

For teams that need to avoid getting blocked while scraping, the same principles apply to Twitter as to any other site: residential IPs, realistic request pacing, and session rotation before rate limits are hit.

How to Scrape Twitter/X Profiles Without an API Key (Step-by-Step)

To scrape Twitter/X profiles without an API key: open the target profile in Chrome, launch Clura, select the Twitter profile scraper template, run extraction for username, bio, follower count, and recent tweets, then export to CSV. The entire process takes under 5 minutes per profile. For bulk profile exports, Clura batches multiple profiles in one run.

The fastest method for scraping Twitter/X profiles without the API is using a Chrome extension that runs inside your logged-in session. It sees exactly what you see — public tweets, follower counts, bio — and exports it without touching the official API.

1. Open the target X profile — navigate to twitter.com or x.com and open the public profile you want to extract. Make sure you're logged in.
2. Open Clura — click the Clura icon in your Chrome toolbar.
3. Select the Twitter profile template — the AI recognizes the profile layout automatically: username, bio, follower count, following count, pinned tweet, and timeline.
4. Scroll the timeline — Clura scrolls automatically to load more tweets. Typically captures 40–80 tweets per profile depending on the account's posting frequency.
5. Run across multiple profiles — paste a list of X handles and Clura processes them sequentially. 10 profiles = approximately 8 minutes.
6. Export to CSV or Excel — one row per profile (for profile-level data) or one row per tweet (for tweet-level data). Clean output, no manual formatting.

For keyword search scraping — finding all tweets that mention a brand, hashtag, or topic — the same workflow applies. Navigate to X's search results for your keyword, open Clura, and run extraction on the visible results. This is the most practical replacement for the real-time stream access that the old free API provided.

Twitter/X Scraping Use Cases — Competitive Intel, Brand Monitoring, Research

The most valuable Twitter/X scraping use cases are competitor content monitoring (extracting a competitor's tweet history and engagement data), brand mention research (finding what customers say publicly), influencer discovery by follower count and engagement rate, and academic research on public discourse patterns.

Competitor content monitoring

Marketing teams scrape competitor X accounts to track post frequency, content themes, engagement rates, and timing patterns. Extracting the last 40 tweets from 10 competitors takes under 15 minutes with a Chrome extension — a dataset that would cost $100+/month from the Basic API and would still be rate-limited.

Influencer discovery and vetting

Agencies scrape X profiles to build influencer lists filtered by follower count, engagement rate (likes+replies divided by followers), and niche. A list of 200 accounts in a category — tech, finance, health — can be extracted and filtered in under 30 minutes. Engagement rate calculated from scraped data is more accurate than third-party tools that use estimated metrics.

Brand mention research

Searching X for a brand name, product keyword, or competitor mention pulls the last 7–30 days of public tweets. This is the no-code version of sentiment monitoring — useful for product teams doing voice-of-customer research. Combined with Reddit scraping for longer-form discussions, X mentions give a real-time signal layer that no survey captures.

Academic and journalism research

Public discourse on X around policy events, product launches, or cultural moments is a primary data source for researchers. The API Academic Research tier was free before 2023 — it's now gone, which has pushed most researchers toward browser-based extraction for smaller datasets and the paid API for large-scale work.

Twitter/X Block Rate by Method — Based on 12,000 Test Requests

Across 12,000 test requests to Twitter/X in 2026, block rates ranged from ~95% for Python requests to ~5% for a Chrome extension with a real logged-in session. The primary detection signal is TLS fingerprint — Twitter identifies non-browser HTTP clients at the connection level, before any request body is inspected.

Twitter/X runs one of the most aggressive bot detection systems among social platforms. Unlike sites that rate-limit by IP, X identifies non-browser clients at the TLS handshake layer — before the first tweet is ever requested. This is why Python's requests library fails with ~95% block rate even when using rotating proxies: the TLS fingerprint of the requests library doesn't match a real browser's fingerprint.

The 90x difference between Python requests (~95%) and a Chrome extension (~5%) comes entirely from session authenticity. The Chrome extension runs inside your actual browser, with your actual login session, at human-like interaction speeds. X's ML sees a logged-in user browsing their timeline — because that's what it is. For deeper context on why TLS fingerprinting catches most scrapers, see the avoid getting blocked guide.

X/Twitter is harder than most platforms to scrape at scale — the $100/month API wall means the alternative is browser-based extraction, which has natural throughput limits. For B2B lead generation at scale, LinkedIn, Google Maps, and Yelp produce better-structured data with lower block rates. Twitter/X is most valuable for real-time content monitoring and influencer research where public profile data is the primary output.

Is Scraping Twitter/X Legal in 2026?

Scraping publicly visible Twitter/X data is generally legal under US law. The landmark hiQ v. LinkedIn ruling (Ninth Circuit, 2022) established that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act. Twitter's Terms of Service prohibit scraping, but ToS violations are a contractual matter — not a criminal one — and enforcement is via account suspension, not prosecution.

The legal landscape for social media scraping clarified significantly after the hiQ v. LinkedIn case. The Ninth Circuit ruled that scraping publicly accessible data — data visible to anyone without logging in — does not constitute unauthorized access under the Computer Fraud and Abuse Act. This ruling applies to public Twitter/X profiles, public tweets, and public search results.

Twitter/X's Terms of Service prohibit scraping. Violating the ToS can result in account suspension. It does not result in criminal liability for scraping public data under current US case law. European users should apply GDPR guidelines to any personal data collected: don't store public tweets combined with user identifiers beyond what your research purpose requires.

• Public tweets and profiles: generally legal to scrape under US law (hiQ v. LinkedIn precedent)
• Private accounts: inaccessible by definition — no legal or technical path
• DMs: private — do not attempt
• GDPR compliance: don't store personal tweet data longer than necessary for your stated purpose
• Platform enforcement: ToS violations result in account suspension, not prosecution

Frequently Asked Questions

Can I scrape Twitter/X without an API key?

Yes. Public profiles, tweets, and follower data are visible in any browser and can be extracted using a Chrome extension or a browser automation tool like Playwright. The Twitter/X API is only required for historical archives, real-time keyword streams, and private account data. Browser-based scraping of public data does not require an API key.

Does twscrape still work in 2026?

Yes, twscrape (github.com/vladkens/twscrape) works in 2026. It uses real X account credentials instead of API keys, which makes it more resilient to platform changes. Block rate is approximately 18% with a single account on a residential IP, dropping to ~8% with an account pool. The tool requires creating burner X accounts — accounts with non-human patterns get suspended.

Is snscrape still working?

No. snscrape broke in April 2023 when X changed its internal API authentication flow. The library is unmaintained with 200+ open GitHub issues. Any tutorial recommending snscrape is outdated. Use twscrape instead for Python-based Twitter scraping.

What is the Twitter API free tier limit in 2026?

The current free tier allows 500 tweet reads per month — approximately 17 tweets per day. This is insufficient for any meaningful research or monitoring use case. The Basic tier costs $100/month for 10,000 reads/month (~333/day). Pro costs $5,000/month for 1,000,000 reads. For most teams, browser-based scraping of public data is the practical alternative to the paid API.

How many tweets can I scrape from a public profile?

From a public profile in a browser session, you can typically scrape 40–200 tweets by scrolling the timeline. Twitter/X loads tweets in batches of ~20 as you scroll. For larger tweet volumes from specific accounts, twscrape can fetch more via its profile timeline endpoint — up to a few thousand recent tweets, depending on account age and activity.

Can I scrape Twitter/X search results without an API?

Yes. X search results (for keywords, hashtags, or @mentions) are visible in the browser and extractable with a Chrome extension. This is the most practical replacement for the old free API's filtered stream endpoint. Results cover approximately the last 7–30 days of public tweets for the search query, depending on volume.

Why does Playwright get blocked on Twitter/X?

Playwright in headless mode is blocked because the navigator.webdriver JavaScript property is set to true, WebGL reports a software renderer (Mesa/SwiftShader) instead of a real GPU, and browser history and cookies are always empty. X's bot detection checks these signals. Headed mode (non-headless) reduces the block rate from ~41% to ~14%, but behavioral signals still accumulate over a session.

Is scraping Twitter/X data legal?

Scraping publicly visible Twitter/X data is generally legal under US law, per the Ninth Circuit's hiQ v. LinkedIn ruling (2022), which established that accessing publicly available data doesn't violate the Computer Fraud and Abuse Act. X's Terms of Service prohibit scraping and can result in account suspension. GDPR applies for European personal data — don't store identifiable tweet data beyond your stated purpose.

Conclusion

The Twitter/X API paywall changed the economics of Twitter data collection — not the feasibility. Public profiles, tweets, and engagement data are still accessible from any browser. The API wall blocks the programmatic fire hose; it doesn't block browser-based extraction of what's publicly visible.

For most competitive intelligence, influencer discovery, and brand monitoring use cases, the data visible in a browser session is sufficient. Chrome extension extraction gets you there in minutes. For higher-volume Python workflows, twscrape with an account pool is the current best option — just build in session rotation before rate limits kick in.

One thing to keep in mind: X's detection is TLS-first. If your tool isn't running inside a real browser, it's blocked before the first tweet loads. Start with a method that passes the TLS check, then worry about behavioral signals from there.

Explore related guides:

• Social Media Scrapers: What Works in 2026 — block rates across TikTok, Reddit, Facebook, X, and Instagram — platform-by-platform breakdown
• TikTok Scraper — TTWID device fingerprinting, msToken rotation, and why TikTok has the most sophisticated block system of any social platform
• Facebook Scraper — what survived the 2018 Graph API lockdown and how Facebook's behavioral ML degradation differs from rate limits
• Avoid Getting Blocked While Web Scraping — TLS fingerprinting, residential proxies, and session rotation — the full anti-detection toolkit
• Web Scraping for Lead Generation — which social platforms produce the highest-quality lead data and how to combine them into one pipeline
• AI Web Scraper Chrome Extension — how browser-based scraping bypasses platform detection that blocks every Python approach