In the competitive world of digital marketing and content creation, understanding your competitors' strategies can offer invaluable insights. One effective way to do this is through scraping YouTube metadata. This post will guide you through building an automation tool to scrape YouTube search results, extract video details, and compile them into a GoogleSheet for further analysis.

The first step is to configure an automation that will scrape YouTube search results. We will use a standard YouTube search URL and automate the process to save and extract video metadata.

To begin, you need to define the search that the automation will perform on YouTube. Using a consistent URL for your search results simplifies the process, but you can also record different type steps for varied searches. The main goal is to trigger your search query within the YouTube search bar and navigate through the results.

Once the search results are displayed, the automation needs to extract the video titles and URLs from the page:

Next, duplicate the tool selection to scrape the links associated with each video title:

The extracted video titles and URLs need to be saved to a Google Sheet for easy access:

With the basic information gathered, you can now set up a second automation to extract further details, e.g., descriptions, channel names, and transcripts.

Begin by setting up a new automation process to handle individual video URLs:

From each video page, scrape additional details:

Many tools can generate video transcripts; for this example, we use Tactic.io:

This step involves copying every line of the transcript accurately and adding it to your data.

Finally, set up the automation to loop through each entry in the Google Sheet:

By following these steps, you can create a robust automation that not only scrapes YouTube video titles and URLs but also fetches comprehensive metadata, enabling you to gain in-depth insights into your competitors’ content strategies. Automating this data extraction process can save significant time and provide a wealth of information for your analysis.

Scraping YouTube metadata can transform how you understand and react to your competitors' strategies. This automation process can streamline your efforts, allowing you to focus more on analysis and less on data collection. Happy scraping!

Step 1- Copy the URL of youtube

Step 2- Click on New Automation

Step 3- Click on Web

Step 4- Click on Continue without cookies

Step 5- Paste the URL of Youtube and click on Save go to URL

Step 6- Click on Scrape a list

Step 7- Click on all the video link name — Select on three dots of Column A — click on duplicate column

Step 8- Again click on three dots of Column B — Click on Scrape Link

Step 9- Click on Confirm

Step 10- Set a google sheet and share with task magic — click on Send

Step 11- Copy the URL of google sheet

Step 12- Click on Send to Google sheets — Paste the URL link — Select sheet 1 — Click on Looks good

Step 13- Select the link from google sheet

Step 14- Click on Rename automation — write the name get videos from search — Click on Save

Step 15- Again click on New automation

Step 16- Click on web

Step 17- Paste the YouTube video link — Click on save go to URL

Step 18- Click on green + icon

Step 19- Click on Scrape single

Step 20- Scrape the channel name in column A - Copy it and scrape the link — click on confirm

Step 21- Click on Click to add a Click step

Step 22- Click on channel name — click on Confirm

Step 23- Select the entire description and click on confirm

Step 24- Click on go to page step

Step 25- Enter the URL and click on save

Step 26-Again click on type to record a type step

Step 27-Paste the YouTube link and click on confirm

Step 28- Add another google sheet with title, channel name and url and copy the sheet link

Step 29- Click on trigger

Step 30- Click on List

Step 31- Tick on use list from a google sheet — Click on setup sheet connection

Step 32- Paste the google sheet Url — Select the sheet 1 — click on looks good

Step 33- Change the loop number — Click on Continue

Step 34- Change the google sheet link to video url — click on Save

Step 35- Increase the loop list range

Step 36- Click on send to google sheet — Paste the sheet link —select sheet 2 — Click on looks good

Step 37- Now click on Play steps and it will export all the data into google sheet as well

Okay. So to build an automation that is scraping some sort of YouTube search result, um, and then we're going to take that search result to another platform that I believe grabs the transcript for that video. So here's how we build this. This is the search that I'm going to be doing inside of my automation.

Uh, we're going to go, we are going to, um, I'm going to use the same URL every single time, but if we wanted to, we could just record a type step of whatever we wanted to search. I'm just going to go to this page instead. And then if we ever wanted to do other searches, we can edit the URL or we can add those type steps.

So, first step of my automation is going to be a web automation. And I'm not using cookies for this because we don't need to log in to YouTube to view the YouTube videos.

So, not using cookies again. Our first URL is going to be that YouTube search that I said we did. So that URL that I grabbed from my normal Chrome browser. And That already has that search being done in the, in the top bar here. Next, what we want to do is we want to take all of the videos here and push them to a Google sheet.

So we're going to use scrape a list, and then we're going to select the title of the video here. And it's important. We do this where we see this kind of box, be this entire portion on the side here. And what this will do is this will find all the other video titles on the page. After we select the first two, again, matching this kind of box that I'm talking about, Uh, try not to go in where it has these green lines tracing in between the text.

Try to keep this outside kind of box that we have here and grab the titles of every video there. You'll see that we have 24 rows now, and these are all of the titles that are displayed on the page. Next, we can click the three dots and then duplicate this. Then the three dots again and scrape links. And now what this is doing is this is grabbing the link to every single YouTube video displayed on the page here.

So I'll confirm that step and that's going to add both of those to our automation. This is all that we need to scrape YouTube to Google Sheets. We're going to have another automation. That takes all of this data and it looks up some more things like the title description, grabs the transcripts and things like that.

So I'll click I'm done and then we need to send this data to a Google sheet. So I'm going to set up a Google sheet. This is just going to be actually I'll use your Google sheet because why not? Okay, nevermind. I only have view only access. Okay. I bought a deals and these are going to be video name, video URL, and then we need to share this with automations at task magic.

com so that we, uh, have access to write to this sheet and to pull data from it. So I'll grab this URL, then back in the task manager gap, I'm going to click on send to Google sheets on the left over here. Then I'll paste the URL. And I can select the worksheet that I want to send this data to. After I do this, I can click play steps and we'll see that this automation is going to go to YouTube and get the title and the URL to every single video that we have there.

Perfect. So this put all of those videos and the link to watch those videos inside of this Google sheet. Now, what we can do is we can loop over this Google sheet to really easily grab the other details that we need and take this data to another website so that we can, um, so that we can generate that transcript and anything else that we may want to do.

So we're going to want to grab one of these URLs to build our automation with before we set up our list trigger. That's going to loop over everything here. So back in the task manager gap, again, I'm going to rename this automation. Since we're done with this, this is get videos from search. We can save that.

And now I'm going to start the next automation in my workspace here. By clicking new automation, web, and continue without cookies. Now, this one that we're setting up now is going to be responsible for taking that URL, grabbing the name, the description, and uh, grabbing any of the other metadata that we need from any other websites that we want to take that to, or anything like that.

So, first step is going to be going to that YouTube video that we copied from our Google Sheet, Then what we're going to be doing is we are going to set up our scrape steps to grab some of the information here. So first step is going to be a scrape single step, and that's going to be grabbing the title of the video again.

Then we can add another column and get the channel author. We can duplicate this and get the link to their channel. And then we can click, uh, let's add a click step of Seymour. Oops. Let's let that load for a second there. So we'll record a click step of this load more button. We, we were running that scrape step before.

Apologies. Confirm that. And that's going to, um, That's not what I meant to click. I meant to click on the description. Apologies. Okay. In the browser, I'm just clicking back really quick to get back to this video. I didn't mean to record a click step of clicking her name. I should have recorded a click step of clicking the more description.

I just kind of forgot what I was doing for a second there. So that expands that. Then we can grab the description here. So I just grabbed, I added a scrape single step and then I'm just tracing the entire block here so that we have all of the text showing. Uh, we can actually go a little bit further in and get just that description here.

I'll confirm this step and that's it for scraping the description. Now what we need to do is go to tactic. io to generate that transcript for us. So I'll use a go to page step, and this is going to be going to one second, this URL of tactic. io slash tools, whatever this was slash YouTube. They have this YouTube tool going on.

Then what we can do is we can, I think

we can upload our transcript in here. But I think it's taking me to one that I already did. One second, apologies. Have not used this website before.

Um, okay, here we go. Apologies. This is the URL that we need to go to. There was something else that was wrong about whatever I was doing. Okay. Apologies for that YouTube video that we went to, we're scraping all the details from that YouTube video. This, we wanted to change this to scrape link, and that's going to get the channel URL.

Then we click the more description. We grab the description. Then we come to this page that allows us to enter the video URL. So we can record a type step of By clicking plus and then type, and then I can click on this input field here and I can paste that YouTube video that we scraped that we are pulled the details from at the beginning.

I'll click confirm. Yes, it is going to be a variable and then a click step of get video transcript. And this will do the submission that we need to actually generate this transcript in tactic. Then we can use scrape single, same thing. We can copy over this entire block here and get it. The transcript line by line I'll click.

I'm done. And then this is it for our second automation. We can dismiss this clicking that now, and we need to set up our list trigger so that we can loop over everything we have in our Google sheet. So to do that, I'm going to add another worksheet here, and this is going to have more details. I'm going to need to add more to this, but I don't know all of them on the top of my head.

We have the title of the video channel name channel URL, and we don't need to add all of these. It'll still put it there. We can name that afterwards. So first step, now that we have this setup is grab the Google sheet URL again, go back into the automation that we just recorded and click setup trigger.

Then we can set up a list trigger. And we're want, we're going to want to say to use a list from a Google sheet. I'll click that and then set up sheet connection and paste the URL here. I'll select the Google sheet that we are pulling records from, which is the first sheet that has all of our video names and our video URLs in it.

I'll click looks good. And then we want to tell it how many times to run here. I'm going to say just five for now, but we can make this number be whatever we want that to be. I'll click continue. And this is the automation. We need to update a couple steps so that we're using variables where we should, and then we are completely good to go.

So we're like 30 seconds away from being done. Okay. So this go to page step here, instead of always going to this YouTube URL, we're going to want to delete that and go to whatever came from our Google sheet, which is the app video URL. This means when our automation runs. It's going to work its way down the list because of the list trigger, and it's going to use the video URL as the first step where we grab, uh, where we go to the first page, which is then going to be taking us to this YouTube video.

Then we scrape the, uh, name of the video channel, channel link, click more description. Then we go to the YouTube generator. Then we want to type the video URL variable. So again, deleting that and then selecting that video URL. And this is the automation we need to adjust our loop. Sorry, to be all the way from the beginning to all the way to the end.

What this means is that for each row in our automation, we're going to repeat every single step. So we're going to run this step, run all the way to step 10. Then we're going to move on to the next row of data and repeat from step one. We're selecting how many rows of data to run from this dropdown here and which road to start on from this input here.

That's going to be it for the automation. I'm going to let a little bit of this play before I stop the video, but basically if we play steps, this is going to do exactly what we told it to do. It's going to loop over all of those URLs in our Google sheet. Grab the details, take that transcript from tactic.

Um, and that is all going to be scraped. I did forget one thing. Um, let me let you see part of this working. We can look at what it's doing in this section here in traffic control, grabbing all these YouTube details. And this is just what the automation is supposed to do. Putting all of this in this data out section, this is going to run for a few seconds.

Um, one thing I forgot for us to do was we need to set up the send to Google sheets. So if I click this and then actually let me, sorry, let me collapse this and then let me click send to Google sheets. Then what we can do is we can grab that Google sheet URL that we're working with again, which I accidentally closed.

Hold on. Okay. We can paste that here. Then we select the next Google sheet to send all of that data to, and we can select that worksheet here. Then we say, looks good when we see our headers and now our automation will export all of that data to the, uh, Google sheet. Now, because I set that up in the middle of this automation running, it's not going to do it.

We can see all the scrape details though. If we click on data out and we can see this is grabbing all of those things we're looking for. Channels, link, description, title of the video, timestamp, or sorry, transcripts, um, all the goodies.