Convert X articles, threads, and tweets to clean Markdown with stable URLs, author handles, and post types in the front-matter — ready to chunk, embed, and store in any vector database.
X content has three structural problems that wreck naive ingestion pipelines. Tweets are short and lose context when isolated; threads are unbounded chains where each tweet only makes sense relative to its predecessors; quoted tweets nest arbitrarily deep. On top of that, x.com renders client-side, so a generic HTTP scraper sees a login wall instead of the post.
Most "scrape and embed" pipelines end up with chunks that are missing context, embeddings that retrieve the wrong sections, and metadata that is too unstructured to filter on. xmd solves the structural problem upstream so your chunker and embedder do not have to.
Each X URL becomes a single Markdown document with structured front-matter:
---
url: https://x.com/<handle>/status/<id>
author: <handle>
type: article | thread | tweet
published: 2026-04-15T12:34:56Z
---
# <Title or first line>
<Body, with quoted tweets as nested blockquotes
and thread sections as ordered headings>Lift the front-matter into your vector store as filterable metadata. At query time you can scope retrieval by author, by post type ("only long-form articles"), or by recency window — without re-embedding.
Tweets are sub-300-token. Embedding each as a single chunk preserves meaning and gives you tweet-level retrieval granularity. Pin the URL into the chunk metadata so citations are exact.
xmd renders thread structure with headings. Chunk by heading boundary so each chunk is a coherent thought. Add a small header summary to each chunk for context overlap if your retrieval evals show drift.
Long-form X articles behave like blog posts. LangChain's RecursiveCharacterTextSplitter with markdown-aware separators handles them well. xmd preserves the heading hierarchy, so the splitter has natural break points instead of cutting mid-paragraph.
The web converter is single-URL and free. Bulk processing — pulling a whole account, ingesting a curated list, scheduled re-runs — requires the HTTP API, which ships next.
Three reasons. First, X aggressively rate-limits anonymous scrapers and changes the markup often, so scrapers break. Second, the rendered HTML carries a lot of chrome (login walls, recommended-posts blocks) that pollutes embeddings. Third, threads and quoted tweets need structural normalisation that a generic scraper does not do. xmd handles all three server-side and gives you Markdown that is already chunk-friendly.
Tweets are short and self-contained — embed each tweet as one chunk. Threads benefit from one chunk per Markdown heading section, which preserves the author's own thought boundaries. Long-form X articles can use any standard recursive chunker (LangChain RecursiveCharacterTextSplitter at 1024–2048 tokens works well) because xmd preserves heading hierarchy.
Yes. The Markdown front-matter includes the post URL, author handle, post type (article, thread, tweet), and timestamp. Lift these into your vector store as filterable metadata so your RAG queries can scope by author or recency.
The web converter handles single URLs today. Bulk ingestion — processing a list of URLs from a CSV or pulling a whole account history — needs the HTTP API, which is on the waitlist. Join from this page to get notified.
Yes. The Markdown output is framework-agnostic. For LangChain, wrap a fetched document in a Document with metadata pulled from the front-matter. For LlamaIndex, use SimpleDirectoryReader on a folder of .md files or wrap each in a Document node directly.
Try a single URL now at xmdbot.com — paste an X link, copy the Markdown, drop it into your pipeline. The structure is what your embedding model wants.