[Ep101] – Google Explains Why Not Every URL Is Crawled And Indexed

by Mar 28, 2022TWIMshow0 comments

Full Transcription

[Introduction] – 00:00

Hey, everyone, welcome to the TWIMshow. This is Sajid Islam your host. And today we are recording episode 101 and this covers the updates from the week of March 21 through 26, 2022.

[Update 1 – Shopify Launches Linkpop & Other Features] – 00:14

So first of all, very quickly, an update from Shopify. Shopify has launched a tool called Linkpop. Linkpop is basically a link in bio solution. This is something similar to what you may have seen from products such as Linktree and things like that, which has been very popular amongst creators where you say “link in bio” and you click on Linktree and then it opens up a page on our website where all it has are the links that you want the person to see in the form of a button and blah, blah, blah. Well, Shopify is taking it a step further. We’re saying if you do that, it integrates with your Shopify store so people you can promote your product or whatever you’re promoting. And then obviously people can complete the checkout all in one interface. Now if you ask me, it’s definitely how it is. It’s definitely all in all solution for Shopify. It helps them to have keep their merchants on the platform and their merchant doesn’t have to buy yet another third party solution, things like that. And obviously a bad news for Linktree because Linktree as a standalone product is now going to be used less and less. Again, I always wondered like if you already have a website. Like for example, our website was is on WordPress and we used to use Linktree and I was like this makes no sense. We have WordPress. Why not just use WordPress and all the analytics are here. Why would I want to pay $10 extra a month for something like Linktree when it makes no sense? We already pay for WordPress hosting. So this is all in all good news. The other two updates from Shopify are Shopify has added two marketing features. One is called Segmentation where it allows you to it’s a predictive feature where it allows you to kind of see who are going to be your high, medium and low customers. Again, it’s all predictive. It’s like me looking at a Crystal ball and saying hey, this guy is going to be this guy is going to be that. And you can kind of know that, right? If you saw someone based on the appearance in your physical store. Now with Shopify, obviously they have much more richer data where there are people. For example, if I was shopping across different merchants on Shopify, Shopify could say, hey, Sajid tends to spend a lot of money. Maybe he’s going to be a high spender. But again, at the same time, guys, it could be misleading as well. So anyway, I don’t want to talk too much about it. I can tell you more and more and why I’m not a big fan of predictive software, but it’s as I’d rather have you spend time on actually doing better marketing efforts. That gets you more better leads and more customers. Okay. The other thing is that marketers can now automate the Shopify customer email campaigns. Users can choose from a variety of email themes and design their own procedures for sending emails and scheduling emails. Basically what this means is that if you’re a Shopify seller, you don’t have to get something like MailChimp anymore. It’s all built in, right. And it also announced that the Shopify email service will allow marketers to send up 10,000 emails for free every month. So again, if you have, say, customers list and you want to send 10,000 emails, that’s fine. Again, Shopify is building in this features, built and whistles to keep their customers on the platform. And of course, if you’re growing, if you send more than 10,000 emails per month, you pay a little bit more, right? It’s a good thing. Win for Shopify. Great win for small businesses who don’t want to really jump from one platform to another. I like that overall. But at the same time, I have reservations around being in one platform such as Shopify. That’s going to nickel and dime you for every little features here and there. That’s for another day. Anyways, let’s move on. Next up,

[Update 2 – LinkedIn Allows Company Pages To Publish Newsletters] – 04:00

LinkedIn is now allowing company Pages to publish newsletters. I mean, again, in the LinkedIn world, company Pages are business pages in the Facebook world, but different platform. Call it differently. But basically now we have a LinkedIn page called Market and Grow. If we’re eligible, we would be ableto create newsletters and publish it and send it to our followers automatically. This is a great features. And followers can also sign up, sign off or sign out. Opt in, opt out. I guess that’s the right word. It’s a good thing. Up until now, LinkedIn last year, I believe they turned it on for creators on the personal profile. Now LinkedIn bringing it into business profile. Obviously, as we have seen with LinkedIn and all the other platforms, when something like this launches, it is always the first core user groups. They have a little bit and after that, they kind of make it available to everyone else. It’s the same thing with this feature, but go ahead and try it out. It’s a good way for you to engage with people who are following your page. By the way, I will tell you, I’m surprised to see that we have more followers on LinkedIn Pages than I expected we would have because LinkedIn is basically used to be an afterthought, but I’ve seen a lot of people started following us. I don’t know how they found us, but they found us and they are staying there. We have a lot more reach on LinkedIn than we have on other platform. That’s just me. Okay, moving on.

[Update 3 – All U.S Instagram Users Can Now Create Product Tags] – 05:22

This update is from Instagram, where all U.S. Instagram users can now create product Tags. What basically this means is that Instagram is allowing anybody, anyone like you, me, whoever is listening. If you’re in the US to add a product and tag it and say, hey, I’m using this product, and obviously the product owner, the business will see that I have tagged that product. So for example, I can tag my Apple iPhone and I can say I’m using Apple iPhone. And this is great for affiliate marketers who can have make deals and say, hey, I’m doing this and blah, blah, blah, buy this, go here, click on my link in the bio and everything and track it in the back end. Right again, business idea. Maybe you can build a software that allows all these tracking functionality, things like that in the back end, but that’s for another day. It’s a great thing. Now, on the flip side, Instagram has also said that the business owner can turn off who can tag your products and things like that. So for example, if I buy a product and it shit and it doesn’t work, I was got scammed. If I tag it and they say, oh, it’s a negative review, they can turn it off and what. So it’s good and it’s bad. I wish Instagram hasn’t done that. It has allowed people to kind of figure out what to do. It’s like I can put up a website that says Apple sucks, right? And Apple is not going to come shut it down and that’s the power of the web. I do not know why Instagram and Facebook as a company has to go and police and moderate everything under the sun. It’s just that they get into this lose-lose situations where they don’t make progress. Right? They could have an appeals process like, hey, if someone’s really trolling you are really saying they are going to have bad things about you, we will review it and we will figure it out. Rather than saying that they’re like saying, well, let’s give the power back to the business and now the scammers are going to all do that, right? You know that we have seen that. Anyways, with that, let’s move on to the next update. Again, in the world of Instagram,

[Update 4 – Instagram Adds ‘Algorithm-Free’ Feed Sorting For All Users] – 07:24

Instagram now adds an option to have algorithm-free feed sorting. Of course, as with everything with Facebook and Instagram and Meta, Facebook is now Meta is that nothing is by default. The good things are not in default. You have to go select the option physically or manually. But what they’re saying is that because of all the things that have popped up last year and prior about how Meta Instagram controlling our view, they’re saying, okay, we will give you an option to basically see your newsfeed the way it shows up by the people you follow or people you favorite. Right? Those are the two options that they’ve added. And the third is the default option is like let our algorithm decide what you want to see. I like this option. It’s just that it’s a step. It’s a bit more involved because you have to remember, oh, I need to go see the way the updates rolled in from the people I follow, people I favorite. But that’s all there is. Nothing else. Then, while staying on the topic of Facebook Meta/Instagram, there is an update from Meta.

[Update 5 – Meta Renames Automated Ad Suite Under Meta Advantage Umbrella] – 08:32

Meta has renamed Automated Ad Suites Tools or Campaign Names or Campaign Objectives in the Automated Ad Suite and they’re calling it Meta Advantage. They have given a different name. Nothing has really changed.
It just made it change the name from Facebook to Meta So they might as well change from whatever they had to make an advantage and things like that. But nothing has actually changed under the hood. There are quite a few changes. I’m not going to cover it in this podcast, but there are things like Advantage Look-alikes and Advantage Detail targeting, Advantage Look-alikes-advantage. What else is that expansion? It just goes on and on and talks about each one of them. What I will do is I will put the link to this update to this Facebook page and Meta page where they announce all these things. You can check it out if you want, but I just want you to know that they have changed their name. So here’s why I’m covering it. If someone were to call tell you, hey, this is what’s called the new name and someone starts off with Advantage. Lookalikes, you don’t get like, huh? Or if someone is still using the old name, they’re not keeping up to date on the news. So I’m not sure what they’re doing because they’re not in the ad system. They’re not seeing things like that. So it goes both ways. That’s why I’m covering it. Okay, moving on. While we’re on the social media

[Update 6 – TikTok ‘For You’ Summit Is Back] – 09:57

TikTok, summit “For You” Summit is back this year. It’s going to be held next month. It’s focusing on the Southeast Asian market. It’s going to conduct a series of sessions with platform experts created to highlight the app’s current marketing potential. The seminar will focus on measurement, content production, and shoppertainment. Right. It’s shopping plus entertainment. Shoppertainment. That’s according to TikTok. What I’m going to do is put the link for the summit in the show notes. If you’re interested, go ahead and check it out. Like, as we always do, what we are going to do is we are going to take the highlights of the summit and share it with you all on this podcast. So if you’re not watching the whole thing and you don’t have to feel like you missed out, whatever happens, the good things I’m going to cover it. The bad things, I’m going to skip it over. Okay, moving on, let’s move into the Google land.

[Update 7 – Google Merchant Center Can Now Be Linked To Google Analytics 4] – 10:48

Google Merchant Center can now be linked to Google Analytics Four, if you’re following our show, you know last week covered how Google has announced the end of life for Google Analytics Three, which is the Universal Analytics. And now they’re moving into Google Analytics 4 the date is still July of 2023. However, you may think you have a lot of time. You really actually don’t have a lot of time. Remember, folks, the best time to repair the roof on your home is when it’s Sunshine, not when it’s raining. Similarly, you want to get ahead of this curve and you want to get GA4 in and you want to go ahead and set it up track, compare, make sure you have all the measurements before you switch over. Okay. But again, coming back into the land of Google Merchants Center, you need to have turned on auto-tagging. and if you do so, then it will automatically go into conversion settings page and connect it to your Google merchants account in GA4 and it should automatically link it up. That’s as easy as that. Okay, next up, we have

 

[Update 8 – Google Has Launched A New Partner Directory] – 11:53

Google has launched a new partner’s directory. Of course, we covered when Google revamped the Partners program in 2022. And we talked about it. And I also went in details about why I don’t like Google being a Google partner. But Google came out and said, you know what? To show our commitment to Google Partners program, we are launching this new directory where you can find all the Google partners. And obviously, these are partners who basically are people who have achieved an optimization score of 70%, spent $10,000 across all managed accounts within a 90 day period, have at least 50% of the designated account strategy certified in Google Ads with at least one certification each product area, search, display, video, etcetera, with a campaign spend of $500 or more, and the ability to dismiss recommendation without them counting against the optimization score. So that’s what it is. That’s the last part. It’s a big one, right? Still, I am not a big fan of this because if you look at Google certification program, it’s nothing but an indoctrination sequence. I’ve taken the Google certification exam. I am Google certified, and I am still telling you that when I took it, it had outdated information. I didn’t think it was valuable. The free master class that I do actually covers
a lot more than what Google certification did. That’s what I say. Anyways, I shouldn’t bash them up anymore because we do Google marketing with that. Let’s move on to the next update on Google.

[Update 9 – Google Explains How They Rank Pages With Abbreviations] – 13:20

So this was pretty interesting. Google basically came out well, not Google. Google’s John Mueller, who is obviously Google spokesperson on the Internet and talks about Google search spokesperson on the Internet. And he talks about how they rank pages with abbreviation. And then he pointed out to a YouTube video that you can see a lot more. But basically the short answer is that they don’t do anything special. And when I say abbreviation, it’s like abbreviations such as Eg or anything like that. Eg stands, for example. It could be anything but basically what Google’s John Muller is saying, hey, we don’t treat it anything special. We just see it and we kind of treat them as tokens on a page. And a token is essentially kind of a word or a phrase on a page that we should just ignore. That’s what he’s saying. They don’t do anything specific. That’s all you need to know. So if you’re worried about a shit, do I need to put Eg or not? Or should I Google? Will I be getting the index or will I get the wrong index or wrong ranking? No, don’t worry about it. Stay focused on other stuff. And if you really want to learn more, I’m going to put in a link to a YouTube video that John Mueller says you watch. It’s back from 2019, I believe, from a Google Search Central seminar. And it talks about a lot of semantics and things like that. That’s only because if you want to geek out, if you don’t want to geek out, please don’t go watch this video. It’s going to be absolutely not the best use of your time. I would rather have you focus on creating something else, like maybe a content, a blog piece or something, email, campaign, things like that. Okay. Next up is.

[Update 10 – Google’s Search Algorithm Has Been Modified to Highlight Better Product Reviews] – 15:02

the Google search algorithm has been modified this month to highlight better product review. And obviously this comes from Google. I don’t make it up. But what basically is Google is saying that, hey, if you want your product review to show up and count for you for SEO, then your product review should have include in depth sorry, let me take it back. Include useful in-depth information such as product benefits and cons, as well as insights on how a product works or how it varies from earlier versions. Come from people who have used the product and can demonstrate what the product looks like now, looks like now or how it is used. Include information that isn’t provided by the makers, such as graphics, audio, or connections to other resources that detail over the reviewers experience. Compare and contrast products to explain how a product differs from its competitors. In general, these are some pretty excellent criteria for the types of review you want anyway. And this is the type of information you want to elicit if you’re trying to call on your top brand supporters to post about your business and products. But by the way, you don’t have control over what your customers or end users or fans write for you. Now, that’s the thing. You don’t have control. But Google is saying this. Do you know who this is applicable for? If you are creating an affiliate site, which I know a lot of people do, a lot of people make a lot of money. Something like the wire cutter, right? If you haven’t been to Wirecutter, go check it out. It’s wirecutter.com. And they are always constantly showing up on page one, position one or position. Two of Google. How do they do that? This is exactly what they do, right. So number one, if you are an affiliate marketer, you can do this. Or number two is you can do it for your own product. Right And you could just contrast with your competitors products. You can write in depth review and create like why your product is better. You don’t have to always necessarily say, hey, I’m the user, I’m doing this. Obviously this is for product reviews. But what you need you to do is flip it on its head and figure out how can I use this? Because if Google is saying this is what’s going to get you indexed and ranked, maybe you want to do it the other way, apply to your own landing pages, things like that. But anyway, if you need more information on how to really write quality reviews or quality product pages, there is a link that Google has shared. I’m going to go ahead and share that in the blog notes. Check it out. Right. It says how to write quality reviews. That’s what it’s like. And it talks about what Google wants you to write about so that you rank on the top with that, folks, the last update of this week is from again, Google’s John Mueller.

[Update 11 – Google Explains Why Not Every URL Is Crawled And Indexed] – 17:45

And he talks about why not every pages are indexed and ranked or crawled and indexed and ranked. Basically, it comes down to the economic problem. What is the economic problem? Number one, he says shares. Well, there’s a lot of junk out there. The number of pages are infinite, right. We have limited resources. The Internet is not great everywhere in the world. Right. And some pages changes frequently, maybe multiple times in a day, such as Amazon’s page. So it’s not possible for us to rank and crawl and index and rank every pages. We figure out a mathematical prediction like where maybe we will rank this page. This page has good information. This page doesn’t have good information. It’s crap. Let’s just ignore it. Things like that, right. So why am I covering this on my show? I’m just covering this because I want you all who are listening to this show to be found. And if you want to be found, basically you need to write good product pages, good homepages, good stuff. So that your Google thinks, you know what, this is something unique. I need to crawl and index this page. Right. You need to start doing that if you don’t do it. If you leave things on chance, especially with things like Wix and SEO and all this stuff. Sorry, not SEO, Wix and Squarespace and everything. People are putting out content out there. It’s like crazy. Not to mention Google also index LinkedIn contents and Google also indexes media content and things like this. It’s pretty busy all day long. So you want to make sure you put out good information out there, good content out there. So that Google thinks you’re worthy of being crawled but what I will do is what I’ve done is in the show notes. I put in the complete answers from John Mueller verbatim so you should check it out if you really want to read what exactly what his thoughts are but basically what this Google is not going to index and crawl everything that you have out there you need to write something good as well as you have to remind Google
to come and crawl okay

[Closing] – 19:48

With that folks that’s it for This Week In Marketing. Now you know everything to be in the know if you’d like to read more make sure you visit our show page and where you will find the links to the articles once again this is signing off until next week, take care, bye-bye.

 

Thank you for tuning in this week, it was a pleasure to serve you all. Hit the subscribe button so that you remember to sign on next week. Same place, same time for another round of This Week In Marketing.

1. Shopify Launches Linkpop & Other Features (00:14)-  Linkpop, a link-in-bio solution from Shopify, enables artists and followers to instantly access and purchase promoted products. Creators on social media networks like Instagram may add the Linkpop link to their profile, and their fans will be led to a Shopify-powered storefront with just one click.

Users who click the link are led to a selected selection of products that they may purchase straight on Linkpop without having to leave the social networking app. Consumers will not have to look for things elsewhere, and marketers will have a better notion of the impact the creator had on the sale.

Separately, Shopify has added a number of features to help advertisers on the site this week.

For example, a new predicted spend tier tool allows marketers to evaluate clients’ spending potential into three tiers: high, medium, and low.

Segmentation. The new customer segmentation tool uses the predictive spend tier as a filter. Geographic location, the amount of orders a consumer has placed, and other distinctive behaviors are among the other divisions.

Marketers can now automate their Shopify customer email campaigns. Users can choose from a variety of email themes and design their own procedure for sending and scheduling emails.

Shopify also announced that its Shopify Email service will allow marketers to send up to 10,000 emails for free every month.

 

2. LinkedIn Allows Company Pages To Publish Newsletters (04:00) – Company Pages may now establish their own newsletters in the LinkedIn app, which will also include new notifications for Page followers for newsletter changes.

In November, LinkedIn brought newsletters to Creator Mode, giving users a more direct approach to tap both into the growing popularity of newsletters as a connection option and optimize in-app interaction.

Company pages will now be able to participate as well.

As previously stated, the ability to notify Page followers with newsletter updates via an automatic, one-time reminder to your audience for each new issue is a major benefit for brands.

Subscribers will also be able to opt-in to get email updates in the future. It could be a terrific method to stay in touch with your audience, and LinkedIn reports that early testers have had a favorable response.

Source: https://www.linkedin.com/pulse/helping-you-grow-your-organic-communities-/

 

3. All U.S Instagram Users Can Now Create Product Tags (05:22) – Instagram will soon allow all users to tag products in their IG posts, starting with users in the United States, as part of its continuous focus on extending its eCommerce offerings. Instagram is now allowing everyone to include direct links to products and companies in their uploads, which was previously only available to certified producers.

This may provide marketers a huge boost in the app for no cost, while simultaneously enhancing Instagram’s eCommerce focus and changing customer behavior.

Users who have a company or creative account can tag products in Stories and Reels as well.

When someone tags one of your goods, you’ll get a notification, and you’ll be able to see all of the marked content on your profile.

At the same time, product tags may be misused, and brands will have control over who can tag their products, as well as the ability to delete tags from their things and brand if they so desire.

According to Instagram, over 1.6 million users tag at least one brand each week, implying that the ability to more precisely tag each product will be an enticing and widely used feature – one that, if it works properly, might help keep the main Instagram feed relevant for a bit longer.

 

4. Instagram Adds ‘Algorithm-Free’ Feed Sorting For All Users (07:24) – Instagram has now made its algorithm-free, chronological feed option available to all users, after introducing it in December as part of Instagram CEO Adam Mosseri’s testimony before the US Senate amid concerns about how Instagram is damaging to children.

To modify the appearance of your main content, you now have two new Instagram feed display options to pick from:

Favorites — This section shows the most recent postings from the accounts you’ve added to your ‘Favorites’ list. You can add up to 50 accounts to your Favorites list, which will help you prioritize updates from these people.

Following — Similar to the conventional Instagram feed, this shows posts from all the accounts you follow in the app.

Both the new ‘Favorites’ and ‘Following’ feeds will display your posts in chronological order as they are published. There’s no more algorithm tampering – it’s the Instagram you used to know and love, with everything in its proper position.

The issue with Instagram’s reverse-chronological feed options is that you can’t set either as your default – you’ll have to manually select your preferred feed presentation every time you check-in.

 

5. Meta Renames Automated Ad Suite Under Meta Advantage Umbrella (08:32) – Meta has stated that it is renaming its ad automation products to make the purpose of each more apparent and to improve performance.

Meta’s ad automation tools will now be grouped into a single portfolio dubbed ‘Meta Advantage.’

Meta’s options will be divided into two product lines under the ‘Advantage’ name. The title change doesn’t provide anything new in terms of functionality just yet, though Meta has hinted at some upcoming changes to its automation possibilities.

Source: https://www.facebook.com/business/news/meta-advantage-automation-ad-suite-performance

 

6. TikTok ‘For You’ Summit Is Back (09:57) – TikTok has launched its second ‘For You’ Summit, this time focusing on the Southeast Asian market, in which it will conduct a series of sessions with platform experts and creators to highlight the app’s current marketing potential.

The seminars will focus on measurement, content production, and ‘Shoppertainment,’ according to TikTok.

This is TikTok’s second For You Summit; the first was held in July of last year. TikTok is aiming to expand its eCommerce capabilities and integrations – hence the ‘Shoppertainment’ reference – and there will be numerous information on the app’s newest usage trends and habits to help marketers plan their tie-in campaigns.

You can register for the For You Summit next month here.

 

7. Google Merchant Center Can Now Be Linked To Google Analytics 4 (10:48) – You may now link your Google Merchant Center account to your Google Analytics 4 property, according to Google. You may also “watch your conversions from free product listings by connecting your Merchant Center and Google Analytics site,” according to Google.

If you already have a Google Analytics (GA4) property, go to the conversions settings page and connect it to your Merchant Center account. This feature will not operate unless you enable auto-tagging, according to Google.

Source: https://support.google.com/merchants/answer/11996548

 

8. Google Has Launched A New Partner Directory (11:53) – The new Google Partners directory, which includes new ways to search for partners as well as an overhauled partner program, has been introduced by Google. “Finding the appropriate partner to help you expand your business is now easier than ever,” Google stated.

What are the new requirements to be a Google Partner? Google settled on the following requirements to participate:

  • Achieve an optimization score of 70%.
  • Spend $10,000 across all of a partner’s managed accounts within a 90-day period.
  • Have at least 50% of designated account strategists certified in Google Ads, with at least one certification in each product area (Search, Display, Video, etc.) with a campaign spend of $500 (or more) in 90 days.
  • The ability to dismiss recommendations without them counting against your Optimization score

Source:  https://twitter.com/adsliaison/status/1506687138204631043

 

9. Google Explains How They Rank Pages With Abbreviations (13:20) – A concern about how Google’s algorithm handles abbreviations was answered by Google’s John Mueller. John went into great detail in response to the question, clarifying that these are simply synonyms and that Google doesn’t do anything extra with abbreviations.

And the short answer is we don’t do anything special with those kinds of things.
We essentially treat them as tokens on a page.
And a token is essentially a kind of like a word or a phrase on a page.
And we would probably recognize that there are known synonyms for some of these and understand that a little bit.
But we wouldn’t really do anything specific there in that we’d like to have a glossary of what this abbreviation means and handle that in a specific way.
So that’s something where, especially when it comes to synonyms, our systems learn these over time.
And for the most part, we handle them when people search and not when we do the indexing.

If you want to learn more, he recommends that you watch this YouTube video: https://youtu.be/DeW-9fhvkLM

 

10. Google’s Search Algorithm Has Been Modified to Highlight Better Product Reviews (15:02) – Google has released an overview of how it plans to improve the quality of reviews displayed in Search results.

In order to provide the greatest review insights to searchers, Google is now highlighting product reviews that:

  • Include useful in-depth information, such as a product’s benefits or cons, as well as insights on how a product works or how it varies from earlier versions.
  • Come from people who have used the products and can demonstrate what the product looks like or how it is used.
  • Include information that isn’t provided by the maker, such as graphics, audio, or connections to other resources that detail the reviewer’s experience.
  • Compare and contrast products, or explain how a product differs from its competitors.

In general, these are some pretty excellent criteria for the types of reviews you want anyway, and this is the type of information you’d want to elicit if you’re trying to call on your top brand supporters to post about your business/products.

If you need more information, Google has also created a comprehensive guide on ‘How to Write Quality Reviews,’ which is in line with what it wants to emphasize in its products.

Source: https://blog.google/products/search/more-helpful-product-reviews/

 

11. Google Explains Why Not Every URL Is Crawled And Indexed (17:45) – Google’s John Mueller published a thorough and open explanation of why Google (and third-party SEO tools) does not crawl and index every URL or link on the internet. He stated that crawling is not objective, that it is costly, that it can be inefficient, that the web changes frequently, that spam and garbage exist, and that all of this must be considered.

Here is what he wrote:
There’s no objective way to crawl the web properly.
It’s theoretically impossible to crawl it all since the number of actual URLs is effectively infinite. Since nobody can afford to keep an infinite number of URLs in a database, all web crawlers make assumptions, simplifications, and guesses about what is realistically worth crawling.

And even then, for practical purposes, you can’t crawl all of that all the time, the internet doesn’t have enough connectivity & bandwidth for that, and it costs a lot of money if you want to access a lot of pages regularly (for the crawler, and for the site’s owner).

Past that, some pages change quickly, others haven’t changed for 10 years — so crawlers try to save effort by focusing more on the pages that they expect to change, rather than those that they expect not to change.

And then, we touch on the part where crawlers try to figure out which pages are actually useful. The web is filled with junk that nobody cares about, pages that have been spammed into uselessness. These pages may still regularly change, they may have reasonable URLs, but they’re just destined for the landfill, and any search engine that cares about their users will ignore them. Sometimes it’s not just obvious junk either. More & more, sites are technically ok, but just don’t reach “the bar” from a quality point of view to merit being crawled more.

Therefore, all crawlers (including SEO tools) work on a very simplified set of URLs, they have to work out how often to crawl, which URLs to crawl more often, and which parts of the web to ignore. There are no fixed rules for any of this, so every tool will have to make their own decisions along the way. That’s why search engines have different content indexed, why SEO tools list different links, why any metrics built on top of these are so different.

Source: https://www.reddit.com/r/SEO/comments/tigvq6/why_seo_tools_dont_show_all_backlinks/