How to Scrape Many Searches Using Variables.
Master Web Scraping with Task Magic: How to Scrape Multiple Searches Using Variables.
In today's data-driven world, gathering information from various online sources has become indispensable to numerous projects and tasks. Whether you're a researcher, analyst, marketer, or simply someone seeking valuable insights, harnessing data from websites can provide a wealth of knowledge. However, manually copying and pasting data from websites can be time-consuming and error prone.
Thankfully, with the Task Magic desktop application, web scraping is easy and efficient! Consider the hours you are going to save! In this step-by-step guide, we'll show you how to use our scrape texts and scrape URL features to pull data from any website and transfer it directly to a Google Sheet. Let's get started!
Here’s a video tutorial on how to do basic data scraping.
If you’d rather read the documentation, these are the following steps. Getting Started - Locating Capturing Steps
You’ll want to create a recording to be able to scrape data from websites.
Login into the desktop application and make sure to select the workspace you want to build this automation in. If you are in a workspace with no automations, then you’ll click the Start capturing button to get started.
If you are in a workspace with existing automations. You’ll find there’s an Add new automation in the top right corner of the application. Click the Add new automation.
Next, click Capture Steps to get started.
Recording Your Automation To Scrape Data - Understanding the Features in the Recording Bar.
After clicking Capture Steps there will be a countdown from 3,2,1 and a chromium window will open. In additional a small bar at the bottom of your primary window will open. We call this the Recording Bar, the Recording Bar will include the features needed to scrape data from any website.
How to Scrape a List of Data
As an example, I searched Yellow Pages for roofing services near me.
To scrape each name, phone number, address, etc. on this list, click the Scrape: List button.
Next, select two of the same elements (ex: two names, phone numbers, etc.) of the list you want to scrape.
IMPORTANT Don’t select ads. They are different than normal elements on the page. It’s important you only select normal elements on the page. 
If you scrape an element that's also a link (like the name of the company that takes you to the company page), you’ll see these two options.
In this example we used both options, text and URL. So, I clicked on the same element twice. The first time was for the text (as you can see on the left), so I selected the “Text” option. The second time was for the URL (as you can see on the right), so I selected the “URL” option.
If you want to scrape the URL links from a website (as long as it is provided), we can do this efficiently with the desktop application.
There is a possible error you can get while scrapping, and it’s the following.
This happens because the two elements you selected weren’t the same (ex: two names, phone numbers, etc). We need to make sure they are both the same in order to make it work properly.
When successfully captured, you can press the small green arrow that is all the way to the right of theRecording Bar and you’ll only see 4 rows of the elements you’ve captured (there's is more than 4 elements, but the view is limited to those 4). If you are satisfied with the elements, you’ve selected you can end the automation.
When you’ve ended the automation, you’ll need to continue to edit the steps because we need to connect a google sheet (this is where the information will be stored).
Next, click on the “Send to Google Sheets” button.
In this case I used a URL. If you go to your Google Sheet, on the top right corner you’ll this share button.
In this option you can see a link icon on the bottom left of the pop up. This button is to copy the link. You’ll also have to share the automation to the following email: automations@taskmagic.com
Once you’ve done those two steps you can continue in TaskMagic. Make sure everything in your sheet is correct. If so, press the “Looks good” button.
Once we have the export Here is where the power of variables cick in. “Manage loop trigger” is a feature that allows you to conect your data. It’s like having a “TaskMagician” that appears and disappears when you tell it to, works the amount you tell it to, and does the work you tell it to.
To find this option, you first need to go to any automation you like.
After you’ve gone to that automation, you’ll see the button on top to the right.
In the “Manage loop trigger,” you’ll see three options:
- “Manage Trigger settings”: Here you can tell the app what time you want it to work for you. For example, if you want an automation to run every day at 9:00 am, you can set that up here.
- “Manage Loop Settings”: Here you can tell the app how much you want it to work for you. For example, if you want an automation to loop 10 times you can set that up here.
- “Manage Google Sheets” here you can tell the app what data you want to work on. For example, if you have a Google Sheet with specific data from a competitor and you want to send an Instagram message to their followers, here is where you’ll put that information.
Let’s first see the “Manage Trigger settings.”
In the “Manage Trigger settings” option, you can choose when you want an automation to run. You can select specific times like each hour, every other hour, each day, every other day, or each week. You can even choose the time zone you want this automation to run in! This feature gives you complete control over your workflow.
In the “Manage Loop Settings” option, you can set the number of times you want a task to be done, let's say send a message. You can choose to send the message to 10 users at a time, 20 users at a time, 30 users at a time, or any other number you prefer. This feature gives you complete control over how many times your automations run and how many users they target.
First need to assign a starting row. In this case, it is row number 3 because that’s where the first username starts.
Next, you need to choose how many users you want to message at a time. In this example, we just want to message 10 users, so we choose 10 rows. If you want to message 20 users, you should modify it to 20 rows.
And last but not least, the “Manage Google Sheet” option. This option allows you to change the sheet that you want to work on. For example, if you want to change the users that you want to message, you can go to this option and select a different sheet that has the usernames that you want to target.
To change the sheet that you want to work on, you just need to choose the new sheet that you want. This will update the users. Remember to use the same header so the automation works properly. In this example, it’s “Username,” so make sure this new sheet has the same variable in the first row.
If you don’t have a Google account connected, you’ll see the following green notice: “To connect by pasting the URL you must share the sheet with automations@taskmagic.com.” It is super important that we share this sheet with this email; otherwise, the automation won’t work.
Once you’ve connected the sheet, you can now press the “Play steps” button.
And you are basically done. Just let our app do its magic.
If you see the “run successfully” notification, that only means one thing.
Go back to your Google Sheet and you’ll see all the information scrapped.
