--- id: "690cdd9a-cd13-4d67-8ecb-377ea6d28115" name: "X bookmarks + media archiving (mechanics)" micro_summary: "Reusable X archiving mechanics: capture live Bookmarks GraphQL headers and walk Bottom cursors with an async in-page pager; capture photos through a page <img> plus CDP Network.getResponseBody; capture videos with yt-dlp against direct video.twimg.com mp4 variants." status: "in_progress" see_also: [] scope: "How do we enumerate and archive a complete X bookmark corpus — posts, quoted tweets, photos and videos — through browser-harness?" --- ## Enumeration - Operation `Bookmarks`, query id `-dgKZ58Dr9YSJYrcgEb5KA`, on `https://x.com/i/api/graphql//Bookmarks` with variables `{"count":40,"cursor":,"includePromotedContent":true}`. - Capture credentials from a live request instead of reconstructing them: patch `XMLHttpRequest.prototype.open/setRequestHeader/send`, force a real request with an in-page SPA navigation (Likes → Bookmarks), then reuse the captured `authorization`, `x-csrf-token` and `x-client-transaction-id`. The captured transaction id survived roughly 130 paginated calls in one session. - Walk the `Bottom` cursor. A page that repeats the final cursor and returns zero tweets is the end signal, not a failure. - The rendered timeline is not the corpus: the DOM showed 4 posts while the API held 1,251 entries across 31 unique pages. - Run the pager asynchronously in the page and poll it from Python. A synchronously awaited multi-page loop exceeds the helper IPC budget (~5 s). ## Media - Photos: inject `` elements (3–4 per round), match `Network.responseReceived`, read `Network.getResponseBody`, base64-decode and write. Page-context `fetch` is CORS-blocked; `http_get` decodes UTF-8 and cannot carry binaries. Sniff magic bytes, because the media URL extension is unreliable. - Videos: take the best `video_info.variants` mp4 at or below 1080p (720p beyond ten minutes) and hand the direct `video.twimg.com` URL to yt-dlp. 134 videos, ~4.4 GB, zero failures. - Quoted tweets arrive inside the parent tweet result and should recurse into the same folder shape. ## Boundaries - Reuse the user's existing X tab read-only; interaction with an active tab requires explicit task-scoped human authorization under the tab workflow loaded at entry. - Session-scoped credentials stay in page context: never write tokens to disk or into tree nodes. --- ### 2026-09-29 01:55 UTC ### 2026-09-29 — re-walk: the capture trick and query id both changed Re-walked the corpus live: 1,276 unique bookmarks (up from 1,251 on 2026-09-21). - Bookmarks now renders under `/i/history` (`/i/bookmarks` redirects there). Query id is now `Glt3WAwBvNSPD-n_sqmX_A/Bookmarks`, a **GET** with `variables` and `features` URL-encoded in the query string, not a POST body. - The documented "patch XHR/fetch, then SPA-navigate Likes → Bookmarks" capture no longer captures anything: a backgrounded `/i/history` tab issues **zero** `/i/api/graphql` requests after load, tab-switching between Bookmarks and Likes serves cached data, and infinite scroll never fires because IntersectionObserver is suspended in a hidden tab. The rendered feed also stalls at 6 articles and never paginates. - Working capture: `cdp("Page.addScriptToEvaluateOnNewDocument", source=)` **before** `Page.reload`, so the fetch/XHR patch is installed before the app bundle runs. That catches the first Bookmarks XHR with `authorization`, `x-csrf-token` and `x-client-transaction-id`. Remove the injected script (`Page.removeScriptToEvaluateOnNewDocument`) when done or close the tab. - Response shape is no longer `legacyResults`: read `data.bookmark_timeline_v2.timeline.instructions[].entries[]`, keep `entryId` starting `tweet-` and take `content.itemContent.tweet_results.result` (`rest_id`, `legacy.full_text`, `note_tweet`, `article`, `core.user_results.result.legacy`, `quoted_status_result`). Cursor from `entryId` `cursor-bottom-*`. - Same-origin page-context `fetch` with the captured headers **does** work for GraphQL (34 pages, 1,276 entries, no HTTP errors, one transaction id for the whole walk). The "page fetch is CORS-blocked" note above applies only to cross-origin `pbs.twimg.com` media, not to `x.com/i/api`. - The header search field (`input[aria-label="Search query"]`) is global X search autocomplete on this build, not a corpus-scoped bookmark filter — it does not substitute for the pager walk. This contradicts the sibling node "UI field is corpus-scoped". - Dump the corpus by polling `window.__all` in slices from Python; 1,276 tweets ≈ 1.0 MB JSONL, well inside the helper IPC budget at 50 per call. --- ### 2026-10-11 23:17 UTC ### 2026-10-11 23:20 UTC ### 2026-10-11 — per-post payload capture and single-file offline render - Full thread + quoted-post capture for one bookmark: open the status URL in a dedicated tab, `Network.enable` on the tab session, then read `Network.getResponseBody` for the `TweetDetail` / `TweetResultByRestId` responses. `TweetDetail` carries the author's self-thread (`conversationthread-*` modules, `items[].item.itemContent.tweet_results.result`) and the quoted post inside `quoted_status_result.result`, so one response yields the whole conversation — no second request needed. - Exception: some status pages never issue those requests at all (observed on `vincentweisser/2108665223293067613` — only `ExploreSidebar`/`getAltTextPromptPreference`); their data ships in the server-rendered payload. Recover the post from the `quoted_status_result` inside any other post that quotes it, or from the rendered DOM. - `self_thread_id` / `in_reply_to_screen_name` + same handle is what separates the author's own thread from other replies; sort by `created_at` then rest_id, not by entry order. - Media needs no CDP trickery for a public archive: `pbs.twimg.com/media/?format=jpg&name=large` and the best `video_info.variants` mp4 (≤720p keeps a 15 s clip near 1.4 MB) both download with plain `curl` — they are unauthenticated CDN URLs. The CORS note above applies only to page-context `fetch`, not to a client-side download. - Offline single-file render traps: re-encode to ≤720p first (~43 MB of X clips → 10 MB), and register each file once keyed by path with the data URIs emitted in one trailing script — quoted posts recur across cards and inline duplication inflates the file. Correction (2026-10-11): native `` PDF experiments were unreliable here, but the pixel-profile inference above did not establish a general Chrome prohibition. PDFs CAN render inside a single offline `file://` HTML. Bundle PDF.js and its worker module as data-URI imports; import the worker into `globalThis.pdfjsWorker` for main-thread rendering, then pass embedded PDF bytes to `getDocument({data})`. Render nearby pages lazily to canvases and use PDF.js's text layer for selection. Tested PDF.js 4.2.67 with networking disabled on a standalone copied HTML: both 16-page and 62-page documents load, first/last pages render and text is selectable. No rasterised page-image archive is needed. An actual module Worker experiment stalled in a background tab; the main-thread worker handler worked. Serialise rendering to bound work.