#PowerPlatformTip 96: 'Scraping Web Data with Standard Connectors'

TL;DR: Fetch a web page in Power Automate without Premium: the OneDrive for Business action Upload file from URL saves the page as an .html file, Get file content gives you the HTML for further processing.

💡 Challenge

Fetching data from specific websites and storing it in Power Automate without premium connectors can be challenging. The HTTP action is Premium. How to do this cost-effectively?

✅ Solution

Use the Upload file from URL action of the OneDrive for Business connector (Standard) to download the page into a file, then read the file with Get file content.

🔧 How It’s Done

Here’s how to do it:

1. Download the page 🔸 Add Upload file from URL (OneDrive for Business). 🔸 Source URL: the address of the page, for example https://www.example.com/prices. 🔸 Destination File Path: a path including the file name, for example /Scraping/prices.html. 🔸 Overwrite: Yes, if the flow runs regularly and should replace the last copy.

2. Wait for the file 🔸 The action reports success after 20 seconds, whether the upload has finished or not. Add a Do until with Get file metadata using path (or a short Delay) and continue only when the file exists, its Size is greater than 0 and, with Overwrite on, its Last modified time is later than the start of the run (store utcNow() in a variable at the beginning).

3. Read the HTML 🔸 Add Get file content using path with the same path. 🔸 If the output shows $content-type and $content, convert it to text with base64ToString(body('Get_file_content_using_path')?['$content']).

4. Extract what you need 🔸 Cut out the part you need with split(), indexOf() and substring(), or parse a table with xpath() (see Related Tips).

🎉 Result

Achieve web data fetching, storing, and accessing using standard connectors in Power Automate. This method is cost-effective and bypasses the need for premium connectors.

🌟 Key Advantages

🔸 Cost-Efficient: Saves on costs by avoiding premium connectors.

🔸 Simple: three standard actions, no custom connector.

🔸 Integrated: Leverages OneDrive for easy data storage and access.

⚠️ Good to Know

🔸 Raw HTML only: the action downloads what the server sends. Content that a page loads later with JavaScript isn’t in the file. Look for the page’s own API or RSS feed instead.

🔸 Public pages only: the action can’t sign in, so pages behind a login don’t work. Sites with bot protection can answer with an error page or a challenge instead of the content.

🔸 Throttling: the OneDrive for Business connector allows 100 API calls per connection per 60 seconds. For many pages, add a delay between the downloads.

🔸 Terms of use: check that the website allows automated access.

🎥 Video Tutorial

🛠️ FAQ

1. How can I fetch website data in Power Automate without premium connectors?

Use the OneDrive for Business action Upload file from URL to download the page as a file, then read it with Get file content or Get file content using path.

2. Can I fetch data from any website using this method?

From public pages that deliver their content as HTML. Pages that load content with JavaScript, need a login or block automated requests don’t return the data you see in the browser. Check the website’s terms of service before you automate access.

3. Why does my flow read an empty or old file?

Upload file from URL reports success after 20 seconds even if the download is still running or failed. Check that the file exists, has a size greater than 0 and was modified during the current run before you read it.

🔗 Related Tips

🔸 Extract HTML Table from Email in Power Automate - parse the HTML this scraping technique brings in, using XPath.

🔸 #PowerPlatformTip 150: ‘JSON & HTML to Markdown with Office Scripts’ - turn scraped HTML into clean, readable Markdown.