Are you tired of manually transferring data from one website to another? Imagine a world where you can automate the tedious task of scraping and migrating data with just a few clicks. In this blog post, we will delve into the fascinating realm of data migration using automation, specifically focusing on transferring information about horses from one site to another.

The process begins with building an automation that scrapes all the horses from a particular website. This involves extracting names, prices, and other details about each horse. By leveraging a Google sheet, the names and links related to each horse are exported for further processing.

Utilizing Task Magic, the automation journey kicks off by visiting the target website and navigating through the pages to extract relevant information. From clicking on specific categories like thoroughbreds to scraping lists of names and links, each step is meticulously recorded to ensure accurate data capture.

The automation simplifies the process by systematically recording steps that involve clicking through multiple pages of listings, ensuring that all desired details are captured. By establishing triggers and loops, the automation efficiently handles the migration of data across various pages, streamlining the entire data transfer process.

To further optimize the automation process, variables are introduced to dynamically adjust steps based on the data being processed. This level of customization allows for seamless data extraction and transfer, empowering users to handle diverse data migration scenarios with ease.

Beyond basic information, the automation extends its reach to extract detailed insights about each horse, including barn names, breeding information, and other key attributes. By incorporating additional scrape steps, users can tailor the data extraction process to meet specific requirements.

As the automation iterates through the list of horses, it continuously refines the data extraction process to ensure accuracy and completeness. By incorporating checks for duplicates and implementing data validation logic, users can maintain a high level of data integrity throughout the migration process.

Looking ahead, the automation framework can be enhanced with advanced features to handle evolving data migration needs. By incorporating smart algorithms and predictive analytics, users can future-proof their data migration strategies and adapt to changing data formats and structures seamlessly.

In conclusion, the power of automation in data migration cannot be understated. By harnessing the capabilities of platforms like Task Magic and Google Sheets, users can revolutionize the way they transfer data between websites. With a robust automation framework in place, the daunting task of data migration transforms into a seamless, efficient process that saves time and eliminates errors.

Embrace the magic of automation and unlock a new realm of possibilities in data migration. Say goodbye to manual data transfers and embrace a future where automation reigns supreme!

Step 1- Click on View more about this horse

Step 2- Copy the link from the URL

Step 3- Click on New automation

Step 4-Click on Web

Step 5- Paste the link in URL and Click on Save go to URL

Step 6- Click on + Icon — Click on Click to record Click step

Step 7- Click on Throughbreds — Click on Confirm

Step 8- Click on Scrape a list

Step 9- Click on Name of the horses one by one to Scrape in Column A —Click on View more about this horse one by one to scrape the link in Column B — Click on Confirm

Step 10- Again Click on click button to record a click step

Step 11- Click on forward arrow — Click on Confirm

Step 12- Click on Dismiss

Step 13- Click on Trigger — Click on List

Step 14- Click on Continue anyway

Step 15- enter the loop no — Click on Continue

Step 16- Name the sheet as Horse — Click on Share

Step 17- Select taskmagic.com — Click on Send

Step 18- Copy the Google sheet link

Step 19- Click on Send to Google Sheets — Paste the Link — Select Sheet1 — Check Column A & B — Click on Looks good

Step 20- We can see all the names and links in the google sheet

Step 21- Click on Rename automation — Name as Scrape all — Click on Save

Step 22- Click on Build — Click on New automation

Step 23- Click on Web

Step 24- Copy the link from sheet — Paste it in URL — Click on Save go to URL

Step 25- Click on + Icon — Click on Scrape single

Step 26- Select the Elements of Horse like Name , Birth year, Height, Color, Adoption and Add in Column A, B, C, D respectively — Then click on Confirm

Step 27- Create another sheet to export the data

Step 28- Click on trigger — Click on List

Step 29- Paste the Link — Select Sheet 1 — Check column — Click on Looks good

Step 30- Change the Loop number and Row number — Click on Continue

Step 31- Click on Go to page — Select @Links — Click on Save

Step 32- Drag the loop from begining to end

Step 33- Copy the google sheet again

Step 34- Click on Send to Google sheets — Paste the link — Select All — Check the columns — Click on looks good

Step 35- Make the loop to 5 times and click on continue

Step 36- Click on Play steps to export the horse details in another sheet

Okay. So we are going to build an automation that scrapes. Oh, scrapes all of the horses from this website. So we'll see here that we have a list. We have a bunch of names, their price, things like this. And then we can click to see more information about the horse with some information, some photos and things like that.

So the way that we're going to build this is we're going to use a Google sheet to export all of the names and the links. To the horse. So like this name and then this link, cause clicking this opens a link here. We're going to export all of that to a Google sheet. And then what we can do is we can loop over that Google sheet, go to every page and grab the extra details that we want.

So what this looks like is if I go and I grab the URL, this is all I'm going to need is this URL that I want to record with. And then I go to Task Magic and I click new automation web, and then we don't need to use cookies for this. We can start building this.

So first step is going to be going to this new vacations website. So I'm going to enter that URL, then click save and go to page. And then we'll see that on the right here, we have now gone to this page. We have all of the horse here. And if we scroll down, we have the next page buttons that we can click to load more of them, which will come in handy a little bit later.

So first step is going to be, as soon as we load this page, we're going to want to click, um, for this particular use case, we wanted to just scrape thoroughbred. So I'm going to click thoroughbreds. Then click confirm and we'll see that that clicks this on the page. Now, next is going to be scraping a list of all of the names, uh, and links.

So I'll click scrape a list. Then I'm going to click on the name here and then I'll click on the second name and we'll see that this finds the rest of the records on that page automatically for us. If it didn't, we could just continue clicking until everything that we want to highlight In that column has been highlighted.

So in this case, it was just the name of the horse. Now I'll click add new column so that we can scrape the link to the horse. I'm going to go ahead and click the first button here, and we'll see that this found the rest of the results on the page. All we need to do is click the three dots here. That's a little hidden.

And then click scrape links. And now we're grabbing the specific link to a specific course. Then I'll confirm the step. Lastly, we only need to record a click of the next page button. That way we can loop over, um, every page of listings here. So click, click, and then I'll record this click step. I'll click confirm and we'll see it do that click step after we record that.

And now we are on page two. So that's it for setting up the scraping automation. As far as recording steps, all we need to do inside of task magic, we can dismiss these actually is set up our list trigger. So to do that, we're assuming, um, that we're going to be having a Google sheet here and this is the Google sheet.

We're going to send all of that data to, um, I guess the list trigger actually plays into both. I'm backtracking a little bit. Sorry. Um, let us set up our. Trigger. Um, and then we'll set up sending to this Google sheet. Okay. Apologies for that. Yeah. First, we're going to be using the list trigger as a really easy way to handle clicking this next page button.

So all we'll do here is click here, click list. And then we can click continue anyways, since yes, we want to use this. And then here for the amount of times we want to loop, we're just going to enter the amount of pages on the website, um, which in that case was four, we could enter five, it's okay if we're wrong here.

We just want to make sure that we, um, enter enough that it clicks through every page that was there. This can be improved to use some other things. Um, but this is a very simple way to do it. Uh, which works perfectly fine in this use case. Next, what we want to do is we want to adjust our loop. So it only repeats the scraping and the clicking of the next page button.

Um, and it does not repeat loading the URL every single time. So to do that, we're going to drag this backslider all the way to the end there, because we always want it to run this next page click. Then the first slider is going to go to the scrape a list step, which is where we are grabbing the name for every horse that's on the page.

So our loop will look like this. We're going to run step one, run step two. Then we run step three, step four, click next page. And we restart from this scrape step, um, which is going to be scraping the next page of results five times because we put five in our loop trigger here, which means it runs these steps.

Five times. So it clicks next page that five times. Okay. So now that that's done, I am going to set up sending this to my Google sheet. So I have a Google sheet here. I'm going to just name this horse for now. Um, and then we need to share this with automations that task magic. com. So I'll just quickly answer that.

And then click send. Now we grab the URL from the Google sheet and we're going to paste this inside of task magic. So I'll click back into task magic, click, send to Google sheets, and then I'll paste my URL here and then I'll select my worksheet and I'll see the headers that I have and the steps that will go to the corresponding header.

We'll click looks good and we are good to go. So now when I click play steps, this is going to go to this website. Click thoroughbreds, scrape a list, scrape the links, click next page, and then go back to scraping a list five times. That way we go through every page of results here.

So there's it scraping the name after it clicked thoroughbreds. Next, it's going to grab the link. I'll just scroll down a little bit so we can see that get highlighted.

And then we click next page. We're scraping the names again. That green border might not show that well on this button, but if we go back into the app, we'll see that there were 12 results scraped for both. Uh, scrape steps there. Next page has been loaded and we will just continue scraping these. I'll just pause until the results are in our Google sheet now, since it's just going to repeat this for the rest of the pages.

Okay. So now that this automation is finished, it ran through all of our steps. It completed the browser closed. Everything was great. Uh, we will see the names and the link to every horse in this worksheet. Now what matters here is that we want to grab all of the details from this page inside here. So this page is what we would see if we had clicked on a horse, um, to see its results.

So if I click view more. Or if I go to this page, it's going to load all of these results. So we're going to set up a second automation that loops over our Google sheet and it grabs the rest of the details there. So first I'm going to rename this just so I don't lose this scrape all. And then we're going to make another automation that loops over all of them.

So to do that, I'll click back into my workspace here, clicking on the left, clicking up here, clicking home, whatever you want to do, and then click new automation. And then not on this one, because again, we're not using cookies here. Now, what we're planning to do is we're going to loop over the URLs from our Google sheet.

So we have a URL like this in our Google sheet, and that's what we're going to be visiting to grab the details. You can see how there's something different. There's a different number for every horse here, which is why it's loading a different horse each time. Or the correct horse for that matter. So to record this, we only need to record it like we're doing this perfectly once.

So I'm going to grab this link so that I have an example to work with, and then I'll click save and go to URL. This step is going to be updated later on to use a variable instead of just always going to this page. I'll go ahead and close these since that doesn't matter. And now I'm back in just my Chromium window.

Which has this specific page opened and I can add my scrape steps to learn more about the horse. So I'll click plus, scrape single, and then I'll click on whatever I want to scrape like this. So I'll click here, click on Nigeria, then click add new column. If it had a barn name, I would want to scrape it.

So we'll record that click step. Now, um, if we really needed to, we could find a horse with this barn name, but, um, this'll work for this example, since it'll know to always grab this box. Then I'll click add new column. We'll get mayor. Ooh, I missed 2021. If we mess up our selection, we can just reclick because we're still working in that same column.

We only need to click Add New Column when we want to move on to the next step. So 2021, Mayor, 16. 2, and whatever else we wanted to grab here that was important. I'll just wrap up with grabbing the city. So that's that. Those are all of our scrape steps. Um, and that's really all we need for this automation.

Um, of course, if we wanted to grab images and things like that, we can just set up additional steps to grab the other details here. So I'll click I'm done. And then all we need to do next is set up our loop that loops over our Google sheet list. Whoops, wrong one again. Our Google sheet list and it goes to every page and then it's going to export all of those details to this other Google sheet that I'm making here.

Um, I'm just going to enter a bunch of empty headers here. Obviously, you should have something more descriptive for your flow. Um, but here we have a bunch of headers, which is going to allow us to export all of those steps I had just recorded. So to set this up, I'm going to click in my trigger. I'm going to select list because that's what I want to build as a list trigger.

I'll continue through that. And now we just set up our Google sheet connection and how many times we want to loop. So i'm going to click use list from a Google sheet since we want to loop over all of these details from this page I'll click set up sheet connection And then i'm going to enter the Google sheet here This is a sheet that maybe we have to share with automations at if it's a different one Um, or if we haven't shared that before but we did that in the previous automation Then we want to read from this worksheet with all of our results.

And we'll see that those headers are there too. And again, we can just double check that, um, by clicking in here. Now back to editing our automation, we'll see that we have a couple new options here, such as which row we want to start at and how many rows we want to loop. So these three settings are really important depending on how you want to run this automation.

If I want this to grab the details for every single, um, item in this Google sheet, then I should be setting it up to loop the same amount of rows as I have here. So this is starting at two and it's going down until 41. Um, so we should make this loop 40 times because the numbers are going to be off by one because of the headers.

So I'll enter 40 here. And what this is going to do is this is going to start at row two, and it's going to run 40 rows worth of information, which ends up stopping us at row 41 there. So then if we wanted to only run, let's say 10 at a time, we could make this 10. And the first time this runs, it'll run 10 times, which will end up changing this number to 12.

After it runs again, it'll be 22, 32, And so on. So this number has the ability to update this number. That way you can move down a list. If you don't want to do that, you can enter two as your starting row here. And then you can say, stop row from incrementing. And this will always run with the same 10 rows of data every time.

And this example, um, we just want to scrape all of these results one time so that we can start building our list. And then we're going to end up having this automation, um, run repetitively, uh, down the road. So right now, just to initialize everything that we need, get all of these results now and get started with something, I'm going to make it start at row two and run for 40 rows.

Now, when we connected this Google sheet, we also added some headers, um, from this Google sheet as variables. So anything from our Google sheet that was in this first row, its data can be used as a variable, which is how we're going to build what we're building here. The first go to page step here, instead of being new vacations.

org is going to be. Uh, links, and then we'll click save. What that does is that makes sure that when this runs and it grabs these 10 rows of data, it replaces it with the links value instead. So the first time this runs, if it's grabbing 40 rows of data. The first run is going to be using this row of data.

If we say at links, it would replace with this. If we say at name, it would replace with this. After it runs all of these steps and it starts its next loop, which we'll go over, it's going to be using this data. And now when we say at links, it posts with this. Or with this for at name and so on until it moves its way down the list now to make the list trigger, do what we want in a loop over the correct steps.

We need to update this to go all the way from the beginning to the end. What this means is it's going to go to the URL from our Google Then it's going to scrape a bunch of different items here. Then it's going to restart from step one with the next URL and repeat these scrape steps. And that's going to allow us to move our way down this list until we finally export all of them to this Google sheet by setting up our send to Google sheet function.

So I'll copy the Google sheet again. I'll click send to Google sheets. So I'm going to paste that there, select our worksheet we want to send to, and we'll see the data that is going to be sent to the correct columns. If I click, looks good, we're good to go. And now we can run this automation. I'm going to just make this loop five times for the purposes of a video.

And then I'll click play steps. So what this will do is this is going to use the first five rows of data from this Google sheet and loop over each URL here until it exports all of those scrape details to my new Google sheet.

All right. I think this is the last one. So we see on the left, it's scraping all these details. This URL is changing after it finishes scraping, which is our step one step running. And then at the end of this, we will see all of those details show up in our Google sheet. So here we go. Um, again, remember that one horse didn't have a barn name, which is why this is empty.

These other ones look like they do. Um, but that is the automation. If we were to run for more loops, which again, I edited right here to be five, this would have grabbed more details instead of just those five. But we can see that the names Nigeria, more than tough, uh, I do what I do, et cetera, is all matching up to what is being scraped on this side.

So that's our automation to scrape all of these horse details. We can set up some additional logic with apps and things like that so that when new rows are added, we can check for duplicates and automatically kind of filter our database to make sure it does.