Introduction

In the fast-paced world of task automation, running into odd situations is a common occurrence. Navigating these challenges and creating effective workarounds can often make the difference between a successful automation process and one that falls short. Today, we explore an intriguing conversation about dealing with automation peculiarities, specifically focusing on effective strategies to keep a browser session alive and minimize repetitive logins.

Understanding the Problem

When it comes to automating tasks, one common issue is the need to repeatedly log in to a website, which can trigger security flags or simply waste valuable time. For instance, in the scenario we discuss, the goal is to keep a browser session running to scrape data at frequent intervals without constantly closing and reopening the browser. Let’s dive into the detailed discussion on finding a solution for this problem.

Keeping the Browser Session Open

To minimize unnecessary logins, the first strategy we explore is trying to keep the browser session open. The idea is to log in once and then continuously run the scraping loop without closing the browser. Here’s how we approached the solution:

Addressing Log-out Issues

As intriguing as the above methods are, we consider the real-world applicability and challenges that may arise:

Creative Scheduling and Randomization

One significant aspect of creating a reliable automation is ensuring that the actions are not easily detectable as automated scripts, which can lead to account bans or security interventions. There are a couple of strategies to tackle this:

Conclusion

Navigating the intricacies of task automation requires both technical know-how and creative problem-solving skills. While dealing with issues like persistent browser sessions and minimizing logins, it’s crucial to think outside the box and employ a combination of strategies. From using Chrome profiles and always-true conditions to implementing randomized scheduling and selective delays, there are multiple ways to enhance the robustness and reliability of your automation tasks. Always keep experimenting and remain flexible—these qualities will lead you to the most efficient and effective solutions.

Step 1- We can login in the command center with login credentials

Step 2- Click on in Progress

Step 3- Click on red Icon to stop the steps

Step 4- Click on + Icon

Step 5- Click on record

Step 6- Click on Setting — Click on chrome profile setting

Step 7- Click on Create profile

Step 8- Click on Profile name and write the name — Then click on Create

Step 9- Click on Launch Icon

Step 10- It takes us to login page of web

Step 11- Enter email address and password to login

Step 12- Click on + Icon and go to filter button

Step 13- Select the options accordingly

Step 14- Click on Setting — Click on Advanced setting — Click on runtime Setting to set the run time of steps or automation

Step 15- Click on three dots — Click on Advanced setting

Step 16- We can add random delay steps from here

Are you seeing the application? Yep. I can see it. Perfect. Great. So, yeah, so I think, um, we've got a lot of that stuff kind of figured out. Uh, the, this is one area. So yeah, I'm just said, um, right now we're just doing this whole loop, you know, each time, every five minutes, do the login, do all these things.

Ideally we could just, Log in once and let it run and just loop on the actual scraping part to start there or to start the selector. Um, but as far as I guess, so are you talking about basically you want it to just stay running in the same browser at all times type of closing? Okay. Problems with staying logged in on the website.

Okay, have you guys used chrome profiles or anything like that with it yet? No. Okay, just curious. I'm going to try to get it so the browser never closes, but I'm just kind of curious just for me to know what people are running into his problems. Um, okay, so what we could do is we could try to build something that's essentially like an always true condition.

What that means is, is we have a step in your automation that Basically is going to, um, we're going to use the filter step to make sure that it always resets back to a previous step. So what filter allows you to say is like, Hey, if step two says Kyle, then go to step two, what we can say is say, scrape something and say, Hey, if this exists, cause we're guaranteeing that it is, cause it's their logo or something, then go to step two.

And now we have an automation that will hit that step and always reset back. Um, that might be one way we go about it. But I'm curious to see now, just so you know, like this run, but by the way, I feel like it's been a little bit. Sometimes I try to get it to stop an active run, and I'm not sure the best way to do that.

Do you want to just turn in the automation off while we're trying to? Yeah, that's cool. Well, let's turn that off. Um, given that automation is running now. Is it at the top of this list? And I apologize as I've never seen someone fill the list like this. I will definitely add a clear all button so that we can remove all we're begging on this, go ahead and click into that one.

That's spinning. And then you can click the stop icon on that. It might be a little slow if that's, uh, like, click that. Yeah, sorry. You can just that entire thing is and then you can click that stop icon. Um, and then that should close that chromium browser. If it doesn't, you can just right click and then force quit that.

Um, when you interrupted with that. Okay. I think I just, yeah, just make it click. There you go. Okay. So yeah, uh, let's, if you want, let's click the plus icon after step one and then record, and what this does, I don't know if you guys have gotten into this. This just relaunches that recording window the same way.

Perfect. And what's the infinite loop condition? Like what, what is that? So what I was talking about, it sounded like you basically want this to continue to scrape and it's going to repeat that with the problem is, is that you'd never want the browser to close. We always want the browser to close. So we kind of have a fighting logic right there of how the app works versus what you guys want is.

To us, an automation is just a simple task of start and finish, so we're expecting the browser to, you know, do step one, do step two, do step three, and then close. You want it to do step one, do step two, do step three, and then go back and keep going back and keep going back, and that's just You want to monitor it, yeah, yeah.

Yeah, so it's possible to monitor, it's just kind of Originally, we weren't going to be this kind of scraping application, and then everybody found it really easy to scrape with our tool. And now we're here and now we're a monitoring application. So there's hacks to do what you want to do. Um, it's just a process to get there.

I guess my concern here, though, is that, you know, it's like some a programmer as well. And so I know when you're fighting against the nature of something, there's challenges. So my one concern is if we if we go this route of, you know, Always stay open, always stay logged in. If something crashes or whatever, there's not, there's no like task monitoring thing to then say, go watch the thing again.

And this is something by definition we want to just have running, you know, all day, every day. And so yeah, yeah, yeah. Most of the day. So my, my, in some ways I'm really wondering if we truly want to fight against the nature. That makes sense. I agree with you on the, and not how to just do the three. What's that?

Sorry, it cut out for a second. I thought you would pause, but that makes that makes sense. Um, I, I agree with you on kind of the problems that could come of that, especially because we don't have like a notify me if it failed type of thing. Um, so what I would assume the best thing to do is, is that we set this up in a way that it will, um, basically run the way we have it run where the browser is going to close and reopen.

So what was was there? What was the reason we couldn't do that? Yeah, we didn't. Yeah, we're just trying to be a bit more stealthy about it and not have it, you know, log in and create a new session every week. We really need to get an update every like 10 seconds. Let's say, okay, from the page, but we don't want it to be constantly logging in.

Um, so that's the issue. I see. So, yeah, so I think Chrome profiles are going to be the way to go with this. What we, so what we set up and I don't, you guys, I don't know if you guys started using it recently or when you guys set up, it's kind of a recent feature. Um, if you click exit recording right here.

Oh, shoot. Yeah. I just realized I did that. How do I, yeah. Okay, cool. Yeah. Just click down there. Um, and then click settings on the top left over there under what's new and then Chrome profile settings. And then create profile, and then just name this whatever you want, like, you know, default, main, whatever, it doesn't matter.

And then go ahead and click create. So what this is, is this is the same as creating like a browser profile in chrome. So the same way you can have a completely different identity, but in a chrome session, that's what this profile is. So now when your automation starts, it doesn't need to do the login.

We're going to have that session stored. So you're just a new you, you're, you're the same, I guess it's a new session, but it's the same user. It's not a new login session is kind of what we could get to here. Um, but you also mentioned wanting to scrape like every 10 seconds. Yeah, that's it. And this brings up something else.

And that is, is that on the, um, in the sign in page, it doesn't store, it doesn't have any cookies. When I went in with the cookie grabber Chrome extension, it doesn't have, does that make a difference? Because this relies on cookies probably to, I mean, they might just be doing, you'd be surprised how bad some websites handle their security of buckets that it might be in the local storage.

I don't know. I mean, we went into that where it's not cookies. They stored it the same way that the Chrome profile would be able to capture that if there's another way, if you're using cookies, then no, you know, you're kind of out of luck if you're not doing that, but like open phone, for example, you can set something in local storage and it logs you into that account.

And it's just like an ID you have to set and it logs you into any account you want. That's the problem with like a lot of these things. They should use cookies for this, but they didn't. Um, I, I think Chrome profiles will be the good way here. I just have a couple concerns with like the tool because you're mentioning performance of like every 10 seconds.

I'm trying to think about the reporting side of that because there could be like some lag there. Um, kind of what, so I guess what, what is this, what is this doing? What is the automation doing? Um, It's just going and grabbing some data that just gets updated randomly and then, um, then we do something with that data on our side.

So we take that process and it's a real time thing. So we're, we're trying to be responsive to the real time nature of the instructions. It's sending. Yeah, just monitoring a DOM element for like an XHR coming in and adding more more text to it. And, you know, obviously not having to log in log out. So real quick.

So when you log in here and say, close the chrome window, if you go back to that site, you have to log in again. Or is it? Are you already stayed stayed at London? Um, that's a great question. Yeah, we got to do that because now if you want to open this chrome profile, we could see exactly how the automation will behave with this.

Um, go ahead and click cancel. And when you hover over this, there's a launch icon. So this, this launches that Chrome profile that you just created. So what we can do is manually log in one time, then we'll close this. And then let's see if you stay logged in, I guess, is kind of the, so I think what it's going to do is it's going to take me and it's, I think it's going to kick me.

So I'm going to go here and then it's going to, what is it going to do? No, yes. Okay. Then it's going to make me, it's going to kick me to MTA, a different website here. And, uh,

oops. Cause even if we don't have to do this stuff each time, that would be nice. Yeah, I mean, basically, I set it up to do this in the automation each time, right? Yeah, I mean, because I understand you're worried about flagging the system because Hans logged in 50 times today. So, the first problem I'm so as far as like someone resumed a session 50 times That's not really that much of a red flag as it is like Hans logged in let alone Hans logged in from five different systems Yeah, right I think so.

We asked you how to put this on the web. So we save that password, um, profile, right? That's just for auto fill, I believe. So we, I mean, it's up to you. It won't really help one way or the other. I don't think. Um, now let's go back to task magic and we can just close this window. So back to task. Yeah. And then it'll say close window there.

And then let's just relaunch it really quick and make sure that when you go back to that, uh, click there's that white text that says close window. Yeah, I was trying to see if it actually. Okay, there I did it. Yeah, cool. So it should look better. I agree. Um, put that launch and let's just make sure you stay logged in.

If you stay logged in here, then your automation instead of having your log in steps, you're just going to say, Hey, use this profile and it will always resume from this session. That's always being logged in. What's the what's the URL we're going to go to again? Is it the MTA or can you try to go back to that?

Yeah, yeah. That, uh, I wonder,

and then this isn't, this definitely isn't something I can just sign up for I'm assuming. Right. Okay. Yeah. Yeah. It's paywall. Yeah. Okay. I was just double checking, um, trying to think about, yeah. So my only, my only concerns are like the. The automation you guys are going for versus like, not that anything our automations have been a really standard, but as in terms of speed, I guess we're not.

I'm worried that you guys won't be excited about the speed of the reporting and I'm worried about kind of the resuming of the sessions here with how frequent you want to check. So, um, just to give you up front about my worries about it. I'm We're not the fastest platform. We're not a speed platform.

We're a get it done type of platform. That's the only difference is, um, a lot of people have set up like trade related stuff and like someone was setting up crypto trading stuff. And I'm like, that five seconds matters. And I just need you to be aware of that. That five seconds is very critical. So, um, I think that there, we might be able to do something like, what is the resolution?

Do you think like, is it like a minute or is it, I'm trying to think, I'm confused why the login wasn't stored. Cause that should have been able to be persistent. Is this normal in like Chrome as well? Do you have to log in every single time? Oh, it's weird. Um, yeah, I don't know. It's okay. Um, I mean, when I open up the main screen that has the data on it, um, I can keep going back to it over and over again.

Okay. During the day. Okay. But when it signs me out or when I, yeah, I don't know. This isn't actually the HDA profile, which created, right? Like that's not. Okay. Um, that would be the, that should be the profile that you did. It doesn't name it, but it should be the same. Yeah, we don't do it. The naming was extremely difficult to do for some reason.

So we just gave up on that. Not many people even know the name shows up there. So we just kind of, but this is when you create that profile. Um, it's kind of weird how to explain it, but basically this will continue. If you create more profiles, they're just going to continue to be person one, person, two person, three.

When you launch them, we'll be launching, obviously based on what you named, things like that. Yeah, but I'm trying to think so we may, I mean, we're going to need to do something kind of fighting the system a little bit. Let's call it like maybe an automation that maybe you create a new log in every hour and the automation can run for an hour, like, right?

That type of, we might have like a little bit of a fight type of thing going. It's really just, we need to get creative with something cause it's kind of weird in all of these different areas. So what you could do is you could definitely let me send you a video really quick of using scrape and filter together the way I'm describing.

So this is an automation that this isn't like extremely similar, but it explains the concept, I guess. What this does is this is scrolling down a page and then continuing to click the load more button if it exists. You're going to be doing this to, hey, scrape this. If it exists, we're going to use the filter to go back to a certain step.

So, I just kind of wanted you to see how you can use something to put you back at another step. Since that's what you're going to need to do here, you're going to want to make sure that you resume from after the login step or whatever, you know, that process is going to be for you. So this is a super, this is just a, I mean, that was just a really quick tutorial.

Just you kind of, I guess, see filter and action. Um, and then you'd probably want to use like the schedule trigger to start every hour. That way also, if the automation fails for whatever reason, the downtime was a max of an hour. Yeah. So basically like the, the flow is, you know, every hour run the set, run the automation inside the automation.

You know, yeah, every scraping every like do this. So, so basically if I come back to. The, um, the steps. Gosh, I always get sure of this. I think this loads the cool. So, basically, we, this is the full, the full thing. We, you know, scrapeless or whatever, um, maybe we should talk about that in a 2nd, but after that, we could say.

Like, wait for some number of seconds, try to do the script. Like, I think I understand basically that, like, stay with this. Yeah, I'll show you really quick. So, like, click, like, for example, add a delay step really quick. Just clicking plus and then there will be a delay. You can add that after step. Oh, you've already added delay steps.

I mean, you're fine there. Um, go ahead and add a filter step. And I'll just kind of show you just to kind of show you what this looks like. Sure. Um, just the plus, and then there'll be a filter right there. And then you can see that then skip to, and if not then else, this is all you're looking for is if you click on one of those options, you'll see that this shows you all the steps in your automation.

You can end the automation. You can resume from a different step, skip different step, just kind of, this is what I'm saying is this is your reset. So that you can continue in the same session for some amount of time, you're going to want to use this so that you can, uh, I guess that that's just this is how I'm going to imagine the easiest way is to maintain the browser staying open.

Yeah, and this is better than the loop. Whatever step type. Yeah. So the thing with what loop is for loop in a technical point of view, trying to explain it in like a way that hopefully makes sense with that is It's meant to loop over elements on the page. So imagine you have a list of followers and for a follower, you want to hit the follow button.

Got it. That's how that works. Or like, you know, like we have all of these lists, like, okay, you guys have all of those tickers that all those calls going out. Maybe you want to loop over each call and send a slack message of each call. That's how you might use a loop. So, um, that's how that feature kind of plays.

I kind of make sense. It's good to know that. Yeah, we have the terminology. Exactly. I'm going to delete that for now. So, so maybe, I think. Um, I think it's worth talking with those scraping a little bit because it seems like we just have to tinker on, uh, you know, staying active within going to, you know, like, basically, like, log in once go do the thing you need to do after you do that, like, click someplace else, click back in, wait a couple seconds, do it again.

Like, we can figure that out. I think. Correct. Yeah. And then 1 thing I want to point out to you really quick, uh, advanced settings on the left over there. Okay. On the like left menu bar and then it's towards the bottom there. Yep. And then run time settings. There's a max runtime here. I will say I don't I genuinely don't know when this has been used.

Can you click that checkbox? Because I don't even remember what you have to set up with this. This is such a okay. This is where you might want to enter like. 64, 000 or whatever an hour is going to be so that we can make sure that this absolutely stops that way we can resume with the next, uh, login. But, uh, that's great.

Test magic here. Yeah, that's awesome. There's a creativity side to it that makes a lot of things fun and possible. Yeah, exactly. Uh, so then, so, so the, the selector stuff, this has been a little bit of a challenge, uh, but yesterday I was able to. Um, get logged in, record the step. I also was my first time using this yesterday.

So it's just learning curve, but I was able to then say, okay, this part of the page, um, scraped list. I clicked on. One, two, three, four examples. That was a nice interface, by the way, and then, and then it figured out, you know, how to do that, which is great. And then I realized that it actually needs to be basically more dibs, more surrounding area from the thing I selected.

And so I don't know, um, what the best way to try to, it's probably going to be to re record that step. Yeah, um, just that's the simplest way. If I had to take a guess. And you're too close. You probably want to remove one or both of those span tags from the selector. Not that I ever want users to do this.

I'm just trying to give you an answer that I think would be it. I would tell every single user to rerecord the step, but you told me you're a programmer, so I'm telling you to remove the span tags. That's kind of just to give you an answer there. No, no, totally. That makes sense actually. And I think. We can after this call, and thanks again for doing this.

Yeah, absolutely. If you can just take off those last, like, probably, like you said, those last cold spans, because I'd rather have more of the data and then on the server side, like, slice out stuff than not. Yeah. So, okay, I can, I should actually Yeah, when you're scraping and you see that green border highlighting things, that's tracing every block on the page.

So, you know, hey, I'm on the span now, now I'm on another span, now I'm on a div. You can, you can kind of go back and forth as much as you want there. Way easier than having to understand. HTML elements, but you've already had the luxury of probably having to go through that. So that's why I'm going to let you know how you could edit this.

At least how you can do it quicker. That's great. Here's how I would do it. But here's how I tell every single other person to do. Yeah, yeah, exactly. No, that, that, um, okay, I'll do that. So you don't get a lot of programmers looking for shortcuts with Task Manager? No, it's, it's really weird. We have such a wide range.

We have, we have people who completely put me on my heels and then we have people who are like, okay, so a URL is what takes us to a website. We have a guy who's a machine learning engineer at Amazon, and then we have that person. It's just, I mean, and it's crazy because both people get value from the tool and both people can learn more from it.

And it's kind of, it's interesting because I will say, usually the engineer has more trouble. Because they want to get into all the settings and we're like, we do it for you. Just let us do it for you. We try to do it all for you. There's one in there. They want to get in the, you know, it's like, no, just like we, we made this for you to not have to do like, I look at that selector.

I'm like, I don't want to go in. Yeah, exactly. Well, that's why we try to stop you. There's an extra pop up to say, just don't do it. Yeah. It's actually funny. It's like, are you sure you really want to do this? Actually, to that point, I don't think we need custom JavaScript anymore, but I did have challenges with that yesterday and I realized.

A couple aspects of it. But, um, so I realized that like you need to return something out of that step. That makes sense. Um, is there any any other like tips or tricks as to let me you I have an article on our JavaScript step that shows a couple of like examples like a API request being done and like a console log.

Like it's just that type of a thing. Uh, let me send you this. How can I, is there a way if I, again, probably don't need to spend so much time on this, but if I am trying to do some custom JavaScript stuff, is there a way to just sort of say, hey, like, in the automation, pause, let me look at the actual, like, inspector, let me look at the console.

That was not obvious. Uh, there is going to be a, um, sorry. I. There's a, I was looking at something else and I saw that there's a pause automation step that you can use in. So it's just, it's a delay, but obviously an Intel, the user tells me to resume. And then what you can do is you can just inspect the console and chromium.

Um, and you'll be yeah. So the custom code that you run is executed from that console. So any console log, you might see some junk from us in there too. But your console log will be in there with all that information. Um, they're just like, sometimes we console like, for example, with like loop steps, we sometimes calculate it based on parent elements, that stuff might get console logs.

So just you might have some other junk in there, just because again, um, We don't tell anybody to go to that screen. It's kind of a kind of a if you got there, it's your fault. Yes, totally. Yeah, that article that I just sent you shows a couple different examples of like performing simple math and returning it doing an API request and returning it.

Yeah, I will say if you're trying to use data from a JavaScript step in another step, you should try to return just like a string. Yeah, for that function. Just it's really complicated to work with objects that are returned from this JavaScript step. Just return a single value and that'll make life a lot easier.

Okay, that makes sense. And again, I think we actually don't need that. Thankfully, hopefully not. We're trying to eliminate it. Yeah, yeah, yeah. So then, okay, we've got, I think the general pattern of run it hourly, try to stay logged in by just going, you know, go back to step two and tab through again or whatever.

Um, the selector should be actually pretty straightforward. Can we randomize that or does it have to be hourly on the hour? Yeah, I, the, the scheduling, I mean, where you can tell we're going as far as like the, how obvious is this thing running? Um, but the, there's no, there's no, yeah. But, but it sounded like you guys wanted to scrape this pretty much 24 seven.

Is that, it's just like, yeah, it's just like seven hours a day, six hours a day. Okay. Is it, and so like is it the same six hours or just like you're just trying to be active? Yeah, yeah. No, no, no. It's the same six hours. I see. Okay. Like open hours. Okay. Um, so do you want it to, like, it starts at six and then it starts at seven 30 and then it starts at, is that kind of what you're talking about?

Or is it just like, you wanna make sure it only runs during those hours? I think it's more, instead of saying 6:00 AM 7:00 AM 8:00 AM 9:00 AM it's like 6:05 AM you know, 7:02 AM I see eight 1:00 AM Right. Um. You probably just wanna add a delay, uh, delay. I don't know if our delay step allows for randomization or if it's every single step has its own randomization.

I do not remember on the top of my head. Okay. Yeah, it doesn't really matter. If you can insert randomizations into it, then you can. So delay, that's gonna be a static delay. But if you go to the first step of your automation, let's go to that one and then let's, uh, no, not the fir. Let's go to the click step two.

And then advanced settings on this and then it will run release. Yep. And then, but I, I want to make sure this is going to be like an up to this amount. So like if you enter 10 minutes, like it could delay for zero seconds. So it's a, it should be a, you know, a completely random scale right here. You could add this random delay step.

What would be a really good way to randomize your login is flood your login steps with these randomized delay, but don't add these delay steps to the inner parts of your automation. Just add it to the login process because then your login will always be off by a couple minutes.