[Proposal] Static machine-readable export of OpenWrt wiki content

[Mark Thurston]

This is a weird email.

"We are highly confident this text was AI generated"



From: Ivan Rainbolt <openwrt-devel at membership.list.ivanthegeek.com>
To: <openwrt-devel at lists.openwrt.org>
Date: Mon, 31 Aug 2026 03:08:52 +0100
Subject: [Proposal] Static machine-readable export of OpenWrt wiki content

 > I'd like to propose providing a static, machine-readable bulk export of the public OpenWrt wiki.
 > 
 > This is motivated in part by the February 2025 "State of OpenWrt wiki and alternatives" discussion, where load caused by automated/AI scraping was discussed and periodically generated static resources were suggested as a way to move work away from DokuWiki.
 > 
 > The existing ToH data seems like a useful precedent. Rather than requiring consumers to repeatedly scrape dynamically generated wiki pages, OpenWrt generates machine-readable hardware data that can be downloaded and processed independently.
 > 
 > Could we apply the same general approach to the prose content of the wiki?
 > 
 > For example, an initial implementation could be as simple as two periodically generated static files:
 > 
 > wiki-manifest.json
 > wiki-current.jsonl.zst
 > 
 > Each JSONL record could represent one publicly readable wiki page and contain fields along these lines:
 > 
 > {
 >   "id": "docs:guide-user:network:wifi:start",
 >   "url": "https://openwrt.org/docs/guide-user/network/wifi/start",
 >   "title": "...",
 >   "revision": "...",
 >   "modified": "...",
 >   "license": "CC-BY-SA-4.0",
 >   "source_format": "dokuwiki",
 >   "source": "...original DokuWiki markup...",
 >   "text": "...normalized textual representation..."
 > }
 > 
 > The exact schema is obviously open for discussion.
 > 
 > I think there are several advantages to publishing the information this way.
 > 
 > Most importantly, automated consumers would no longer need to crawl thousands of individual DokuWiki pages. A generator running locally on the server could read the underlying DokuWiki content periodically, produce a compressed artifact, and let the web server/CDN handle downloads as an ordinary static file.
 > 
 > It would also provide a supported interface for more than AI systems: offline documentation, full-text search, archival and mirroring, research, third-party OpenWrt tools, local documentation browsers, and RAG/search systems could all consume the same artifact.
 > 
 > Keeping the canonical page URL in every record would also allow tools using the corpus to continue citing and directing users to openwrt.org as the authoritative source.
 > 
 > I would suggest keeping a first implementation deliberately simple. A complete snapshot regenerated perhaps once per day would already eliminate most reasons for automated consumers to crawl the wiki. Incremental change files, media manifests, or more sophisticated representations could be added later if there is demand.
 > 
 > I also think there may eventually be value in doing something similar for the public portions of forum.openwrt.org. Discourse already stores topics and posts structurally, so a public-content export could preserve topic/post/reply relationships much better than scraping rendered pages. However, that raises additional questions around user content, deletion, privacy, and policy, so I think it makes sense to treat that as a separate follow-up discussion rather than expanding the scope of this proposal.
 > 
 > The immediate proposal is therefore just:
 > 
 > DokuWiki public content
 >           |
 >           v
 >   scheduled generator
 >           |
 >           v
 >   wiki-current.jsonl.zst
 >           |
 >           v
 > search / archival / tools / AI / RAG
 > 
 > with normal HTTP crawling of the wiki discouraged for consumers that can use the bulk dataset instead.
 > 
 > If this approach seems useful and acceptable in principle, I'm willing to help prototype the exporter and work out a minimal schema and implementation.
 > 
 > I'd be interested in thoughts on whether this fits with the direction discussed previously, and where such a generator would best live within the OpenWrt infrastructure/projects.
 > 
 > Thanks,
 > Ivan
 > 
 > _______________________________________________
 > openwrt-devel mailing list
 > openwrt-devel at lists.openwrt.org
 > https://lists.openwrt.org/mailman/listinfo/openwrt-devel
 >