Python AI Scraper: Clean HTML and Respond in Markdown

A guide details the construction of a web scraper in Python that does not return the raw page but a targeted response in Markdown. The process cleans the HTML, converts it to Markdown, and queries a lightweight OpenAI model, with the explicit goal of improving readability and saving tokens.

The LLM Responds Under Constraints and in Clean Markdown

The function answer_query_from_page encapsulates the logic for querying the model. The prompt assigns the model the role of a scraping assistant and provides it with two inputs: the Markdown extracted from the page and the user query. It imposes a strict framework: return only clean Markdown, rely exclusively on the content of the page, and refrain from inventing missing elements. Navigation links, buttons, CTAs, popups, decorative labels, image captions, and repeated marketing fragments must be ignored, with explicit examples like "Start for free," "Contact Sales," and "Your AI Agent." Conversely, the focus is on headers, paragraphs, product descriptions, feature sections, pricing details, documentation text, and factual claims. If the page does not contain the answer, the model must respond with "The page does not contain this information," all while remaining brief, clear, and targeted. The call is made via client.responses.create with the defined model, and the function directly returns response.output_text. This Markdown format facilitates display in a notebook, saving to a file, and reuse in other AI processing chains. This is the step where the scraper truly becomes useful.

Reducing Tokens and Focusing on Essentials Rather Than the Entire Page

The core of the method consists of not sending the entire page to the model. The content is first cleaned and then converted to Markdown before being utilized to respond to a specific query. This approach decreases token usage and produces a targeted output in Markdown rather than a return of the entire page. The expected benefits are a clearer, more readable response that is easier to integrate into other workflows. The operational goal is explicit: read the cleaned content, understand the query, and return a relevant answer. A smaller model, defined here as "gpt-5.4-nano," is preferred since the task does not require complex reasoning. The approach thus assumes a deliberate sequence: clean, convert to Markdown, then query the model.

Preparing the Environment: Packages, API Key, and Model

The project is conducted in Jupyter Notebook to validate each step before turning it into an API or application. The required dependencies cover HTTP retrieval (requests), HTML cleaning (BeautifulSoup), Markdown conversion (markdownify), text correction (ftfy), model calling SDK (OpenAI), and secure loading of the key (python-dotenv). The corresponding modules are imported, including Markdown and display for rendering in the notebook. The OpenAI key must be provided as an environment variable, ideally via a local .env file containing OPENAI_API_KEY=your_api_key_here, after which the client instance is created by reading this variable. A check raises an explicit error if the key is missing. On the account side, billing must be set up; prepaid credits may be necessary on a new account, and it is possible to select another model if the chosen one is unavailable. The project sets MODEL_NAME to "gpt-5.4-nano," a choice consistent with a task that does not require complex reasoning.

Fetching the Page with a User-Agent and Network Safeguards

The function fetch_page performs the HTTP request with an explicit User-Agent header to signal the origin and avoid blocks that some sites apply to clients without a User-Agent. A timeout of 15 seconds is set to avoid waiting indefinitely if the server does not respond, and raise_for_status terminates execution in case of an error code, such as 404 or 500. The function returns the text body of the response. The guide illustrates a real call targeting https://www.olostep.com/ and displays the first 500 characters of the raw HTML, allowing one to observe the amount of extraneous elements present before cleaning.

Cleaning the HTML: Removing Scripts, Menus, Forms, and Popups

The cleaning step begins by correcting any encoding issues, followed by analyzing the page with BeautifulSoup. Tags considered noisy are removed: scripts and styles, graphic and embedded content, navigation elements, headers and footers, sidebars, forms, and buttons. Beyond these obvious cases, an additional pass scans the tags to detect classes or IDs containing typically irrelevant terms like popup, cookie, navbar, newsletter, or modal; these elements are collected and then safely removed. The function returns the body of the page when it exists, or the HTML tree otherwise, to provide a cleaner basis for conversion to Markdown.

Converting to Markdown and Filtering Recurring Scum

The conversion is performed with markdownify in ATX header mode, followed by a second pass for encoding correction. The Markdown for images is removed using a regular expression to retain only useful text. The processing also eliminates unnecessary spaces and compacts line breaks to further reduce the volume sent to the model. A small list of recurring strings, such as confirmation messages or form error messages, is filtered by ignoring these lines after normalizing to lowercase. The remaining lines are joined to form a compact and readable Markdown, with headers, paragraphs, and genuinely informative lists.