What is the Future of SERP Tracking?

Faced with the dilemma of complying with Google’s terms of services or continuing to provide Internet marketers and website owners with keyword ranking data, Raven Tools chooses not to allow their subscribers to track SERPS any more.

So, it looks like Raven users will either have to find a new rank tracking tool or find new ways to track the effects that rankings have on a website’s traffic volumes. Either that, or just go out to find another tool that will continue to provide ranking reports instead.

As a search engine optimization professional, myself, this whole incident brings up some important questions to consider. What is the future of SERP Tracking and reporting rankings to website owners, Do we continue to focus on search engine rankings? What do we do?

If you are like me – which, as a business owner or Internet marketer, I know you are – you are still concerned about organic search engine rankings, So, is there another way to continue tracking and monitoring your website’s position in the search engines? For instance, there is a way to get the position a search results was in Google when it was clicked. We could create a secondary profile in Google Analytics for each client and use a custom filter to record the position that the keywords was in when the search result was clicked.

Google Webmaster Tools' Average Position of a Keyword

Google Webmaster Tools’ Average Position of a Keyword

Or perhaps we’ll have to use Google Webmaster Tools to get data somehow. In GWT, you can see the “Average Position” of a keyword over a few weeks.

As we’ve discussed, ranking data is still a great way for Internet marketers to show their clients if they are gaining or losing visibility in the search engines. It helps SEO’s to report how effective their efforts have been.

So, back to the dilemma. Because Raven Tools uses Adwords API data to show the keyword search volume and advertiser competition of a keyword, they can not use scraped ranking data. But, if another rank tracking tool out there doesn’t need to be in compliance with the Adwords API TOS (because they don’t use that API to show search volume and advertiser competition), they can still technically provide their ranking data and not be violating any TOS, right? Perhaps. I’m not completely sure.

Raven was faced with a decision. This was a tough business decision, according to an open letter by Patrick Keeble, CEO of Raven Tools.

A message from Patrick Keeble CEO of Raven Tools about no longer supporting SERP Tracking because their data comes from scraped Google data.

However, Keeble goes on to say that scraped Google ranking data provides unreliable results, inflated customer expectations (or inflated customers’ clients’ expectations), is risky to use (because it violates certain Google TOS), has an unpredictable future.

Was it really a tough decision for Keeble to make? It sure seems like his company weighed the pros and cons. The tools they provide rely heavily on data from the Google Adwords API. They have a lot of research tools to look up data on keywords and they also have an interface that allows you to view your Google Adwords account performance right in the Raven dashboard. Yes, Raven had to make a decision, and as discussed earlier, it was a business decision. They made a good case for the importance, reliability, and future of using keyword rank tracking as a metric, but is the SEO world ready to let it go just yet?

Keeble says,

“This metric, though deemed essential by some SEOs, would only continue to diminish in relevance as campaign performance metrics replaced it in importance.”

It sounds like Raven’s future will be more centered on providing tools and data that prove ROI, conversions and goals and not simply rankings.

In a private conversation I had with, Raven Tools’  CPO, Jon Henshaw, and a few of their team members, Jon says that their plan is to eventually incorporate the Google Webmaster Tools “Average Position” data into their tool, but the GWT nor the Google Analytics supports that functionality yet.

Raven Tools’ CPO, Jon Henshaw, mentions their future plans.

Also, Raven Tools’ CTO, Scott Holdren, says here in the GWT product forum, that the Google API is not yet providing the data we would need in order to grab data from Google Webmaster Tools to use in third-party reporting tools.

Scott Holdren, CTO of Raven Tools

Scott Holdren, CTO of Raven Tools discusses the state of Google’s API.

In another discussion, Scott Holdren says,

“A temporary measure could be to manually import CSV data from GWT, but long term it needs to be data available from the API. The Google Analytics team has confirmed that despite GA having access to Average Position, the data still lives in GWT.”

He says that the Bing Webmaster Tools API actually does include Average Position data, but that it is not as desirable to Internet marketers and webmasters as the Google data.

In reply to a discussion on SEOmoz’s Q&A where I asked, “What’s the future of SERP Tracking?” and “Is SEOMoz’s SERP Rank Tracking in compliance with Google Adwords API Terms of Service?”

Marketers are skeptical when it comes to some of Google’s actions at times. My colleague / acquaintance, Greg Huegerich says,

“The real question… does Google seek to monetize this through their own tools and interfaces? Step 1 is to eliminate the current solutions, Step 2 is to step in as the market savior with their own paid tool offering.”

As any marketer might, Greg Huegerich wonders about Google’s true motives.

While I’m not sure that is Google’s agenda in this case, it is interesting to think of why Google is so protective of their ranking data in the first place.

Maybe we never should have thought in the first place that we had the right to do bulk lookups in an effort to determine where our websites rank in the Google search results for certain keywords? Or maybe we just need to take our focus off of rankings and put it on other metrics like conversion rates and traffic volumes instead like Jon Henshaw says.

Many search industry professionals are upset. Milind Mody, CEO at eBrandz – an SEO agency in, India – says,

“Scraping SERP data is against Google’s TOS and they have not given permission/ rights to any company. The situation with Raven really stinks. In my opinion it is Google using its dominant position in search to enforce their rules.”

Gyorgy Bolla, Senior Online Marketing Consultant at First Rate replied to my question here, saying that it appears SEOmoz has “no plans to change their data service options”

With the question of “What’s the future of SERP Tracking?”, a lengthy discussion broke out on the SEO Club of SE Wisconsin‘s Facebook page.

The article posted on the Digital Always Media Inc. blog, Jim says,

“The announcement from Raven Tools earlier today marks one of the last major milestones in the transition of the industry. The old way of reporting results is now effectively dead. Google has made it clear that serious analytic companies will no longer be able to provide ranking reports using data scraped from Google by query-bots. Keyword Ranking Reporting will be yesterday’s news within the next twelve hours. It’s a natural maturation but it’s a big one. One which bears marking.”

In response, one Wisconsin-based SEO, David, says,

“I am so tired of these articles. Because Raven is dropping rank tracking, it has NO impact on SEO or the future of SEO or the “maturation point… I’ve been doing SEO for 10 years, every year it was always something new to be the “death of seo” or some variation.”

Although David is in a similar situation as the rest of us, he doesn’t seem as concerned. He states,

David Kauzlaric says, “Nothing has changed.”

“Nothing has changed, though. Just because Raven is shutting down their SERP tracker, there are 50 companies taking their place. Not one thing has changed.  Clients dont care about content strategies, they care about results. If content is required to get them results, then so be it, but they only care about results. The idea behind SEO is to rank websites for keywords, is it not? I don’t know where [edited: person] is getting his clients, but I’ve done over 500 SEO campaigns and I don’t recall a single client out of the 500 that didn’t care about keyword rankings.”

Another Milwaukee area SEO, Mike, says,

“I think my life is over…I rely on SERP reports from Raven because…well 1) it’s our contracts and 2) IMO, it’s a good indicator if your making progress… And moving a ton of customer reporting to a new service scares me, lol.”

This brings up an interesting question in my mind. Should we ask Raven Tools for a discount on our subscription fee since their tool will no longer do what we signed up for it to do? I agree with Mike in thinking that moving over so many clients to a new rank tracking tools and recreating all of those reports is scary. But, should we even bother going out and looking for a new SERP tracking tool? Are Internet marketers ready to give it up? Are our clients going to be upset when their SEO company can’t or won’t provide them with this data any more?

Another fellow Internet marketer has a different stance. He says,

“As far as rankings go, we’ve always pushed our clients away from those as numbers to follow, so dropping that as a metric doesn’t hurt us and our reporting at all… Does suck though overall given that it’s been a cornerstone of the product… It will be interesting to see how SEOmoz is affected depending on their integration and data sources.”

“…what if reporting on rankings wasn’t business as usual? I think it’s an easy ‘sell’ to say that it’s highly important and should be focused on, but it doesn’t really provide the most value. I don’t (personally) believe that rankings are largely important and educate our clients that way. I think this is a problem mostly for those that did make it a highly important success metric, which yes, it is going to suck for them…”

So, what’s the future of SERP Tracking? Damned if I know! Tell me your thoughts?

This entry was posted in Ranting and Raving, Search Engine Optimization. Bookmark the permalink.

41 Responses to What is the Future of SERP Tracking?

  1. Focusing on keyword rankings is not enough to define success in a strategy for a client. Conversions, increased organic traffic, referrals, and ROI always need to be apart of an SEO strategy. With algorithm updates happening so fast, and ever changing, how is a tool that does the ranking reports supposed to keep up? While it’s aways great showing a client Page 1 position 1, and having a tool report that for us busy SEOers we probably need to focus our efforts in other measurements of success.
    As for Google, when people ask what I do for a living, often times I want to reply “I work for Google” which is not so far from the truth.

  2. Russ Offord says:

    If anything, this move of Raven’s away from SERP rank tracking will only help motivate the SEO industry (at least those of us who relied solely on Raven for tracking/reporting rank) to find and report other valuable KPIs.

    We can all go find a new rank tracking provider, like http://www.linkdex.com, or similar… and pain-stakingly re-create all our our client’s keyword/competitor tracking stats… but perhaps along the way we’ll find a way to lessen the need to report ranking and focus moreso on other meaningful data.

    However, rank tracking will never ‘go away’ completely… like Scott said in the SEOWI Facebook discussion: “… people will continue to wonder what position they’re in and we’ll continue to look for new ways to show them how they compare to their competitors online.”

    Russ Offord

  3. At the end of the day…This is a MAJOR change to RavenTool’s products and services. I have yet to read a community tweet/blog post/comment supporting this. I really hope they decide to tell Google they don’t need their stinking Adwords API access.

  4. It’s a sad day for SEO. Google taking more out of the value of ranking. SEO is going more towards personalized search, however, ranking is still key to 99% of SEO’s goals.

  5. Jon Knebel says:

    I love what David Kauzlaric says. It is going to continue to be ridiculous for a long time to think that rankings don’t matter. I love that rankings are becoming more diverse with different types of results and schema, and I love the changes to the alogorithm that Google has made that, IMO, are encouraging better content on the Internet, forcing our marketing pursuits to diversify and become more subtle, but, there’s never been a time more than now that consumers are trusting search results–esp. those near the top.

    • Scott Offord says:

      You’re right, Jon. Google’s changes are “encouraging better content on the Internet”. Ultimately, it is better for them and for the users of the Web.

  6. the problem is.. avg is a flawed metric…

    if you want to properly document rankings you have to do it in a “clean room”

    no personalization
    NO geolocation.. except country.. use “united states” as location…
    (for non intl seo.. if intl use .com with that country as location which will change to appropriate country.. i.e. in Milan, .com gives you ,it results… )

    (trust me.. friend of mine had issue with bungled google.it search for “used cars brooklyn, new york “..)

    no toolbar, no gmail logins in another tab… clear cache.. turn of chrome ext.. pws=0 search…. clean…. no facebook, G+ or twitter either ..

    Why is ranking important..?

    Google your favorite brand,,,,

    Are they #1?

    Do They have a PPC ad?

    Would you have clicked on the PAID ad instead of the FREE #1 organic listing?

    (Why are they paying for a click on their brand name when they are #1?)

    (in advance, the answer for “what would you put there?”)
    (whatever your new products and services are you want to make customers aware of, otherwise.. please click on the #1 listing FREE link below.)

    and thats just the beginning of why #1 is important.. there is over a decade and a half’s worth of data to prove being #1 for the right term, at the right time in the buying cycle, the exact moment when that customer is ready to purchase.. $$$$

    it’s the other terms.. the “i would not shop at.. don’t ever go to….” that you need to get finished getting the easy #1′s out of the way…

    do these first.. then worry about the hard ones..
    the brand
    the brand + locations
    the brand + every single possible piece of intellectual property the brand owns…
    anything client has that is currently providing qualified leads of some kind/sales/in-store visits, etc..

    p.s. who says you can’t rank in personalized search? don’t you have a social team too?

    • Scott Offord says:

      Steve, you make a good point. Social is a great way to start permeating the personalized search results. Social has become an important part of the search engine optimization process! Scott

      • Dana Tan says:

        Hi Scott, Great write-up on the whole issue. Some of the comments here have speculated that Google is aiming at eliminating the competition and then jumping in as the “savior” with their own paid tool.

        Some say that’s not Google’s goal. I wonder.

        To continue Steve’s proposal that permeating personalized search via social will become the predominant strategy for SEO success….don’t you find it extremely interesting that Google just acquired Wildfire?

        Things that make you go “Hmmmmm…..”

  7. This is far from the death of SERP tracking–there will always be companies willing to provide scraped rankings. I do think however that it will lead to more SEOs running their own trackers (AWR, for example) instead of relying on SaaS products for that functionality.

    Perhaps Big G will improve the WMT reporting and that will be enough, but SERP rank is one of the metrics that clients always want to see and that need will continue to be filled by other market players.

    • Scott Offord says:

      Jesse, here’s hoping that Google will improve their SERP Average Position functionality! Thanks for your feedback. Scot

  8. Mark Boeder says:

    I’m not an SEO professional but my $.02 worth … I agree with Jessica Dunbar … “Focusing on keyword rankings is not enough to define success in a strategy for a client. Conversions, increased organic traffic, referrals, and ROI always need to be apart of an SEO strategy.”

    My belief is that keyword rankings became very important because that is many times the extent of the client’s attention span and understanding. It is a relatively simply concept they can quickly get their mind around (and see). However, and it is more complex, more difficult and more selling is involved to actually get the client onboard and involved with a more comprehensive strategy and reporting that comes with defining and tracking “conversions, increased organic traffic, referrals, and ROI”

  9. Adam Parikh says:

    Thanks for taking the time to write a great piece on this issue Scott.

    While I agree ranking is not the most important SEO metric, I feel there is a pattern developing and it has me concerned. I’m still dealing with the fallout of ‘keyword not provided’. Missing the user behavior of what I have observed as roughly 20% (and climbing) of organic keywords not made available is unsettling.

    Now another avenue of data is being shut off. The cynic in me sees the future Greg H. maps out where there will be a Google Webmaster Tools Premium with various levels of data at different price points.

    As a publicly traded company, Google is beholden to their shareholders now and the people that make a living in SEO.

  10. Scott Krager says:

    Great article Scott, lots of nice insights here.

    In my view most of Google’s changes the past year or two stem from a desire for Google to be more in a direct line to the small business pocketbook. Google giving away websites to businesses, more features in Google Webmaster Tools, more Google sales reps, etc.

    The issue I have is that many small businesses owners don’t want to learn about Google Webmaster Tools, or Adwords, or (not provided)…they just don’t care. They want to be able to hire or work with someone, usually local, that they can trust to handle this “online marketing” thing for them. And I think this is what Google just doesn’t understand. Not every business wants to learn this stuff that we love.

    Sometimes middleman can add value to an ecosystem, and in the case of ethical SEOs and PPC experts they add a ton of value to the businesses they serve.

    • “The issue I have is that many small businesses owners don’t want to learn about Google Webmaster Tools, or Adwords, or (not provided)…they just don’t care. ” – This is understandable from a clients perspective.

      “They want to be able to hire or work with someone, usually local, that they can trust to handle this “online marketing” thing for them. And I think this is what Google just doesn’t understand. Not every business wants to learn this stuff that we love.”
      Amen, and thank you google for the job security.

      I look at search over the past year or two, google is personalizing search, on an individual & geo location. That’s awesome. We need to adapt, and grow with the change. Google isn’t serving only business’s here. There are many more “users” searching, and that seems to be the focus for them.

      I just absorb as much information as fast as I can. LOL is their any other way?

  11. Milind Mody says:

    Nice writeup Scott.

    I do not think Google will introduce GWT Premium.

    Google is very sensitive to other tools scraping their SERPs and somehow using that data to either reverse engineer their algorithm or track historic changes for major SERP terms.

    (Not Provided) really sucks. And now Google wants people to force SaaS companies to stop reporting SEO rankings. And how many major SEO updates did we have this year ? See the direction where we are going?

    As someone else pointed out, Google is the largest Scraper in the world. So isn’t this hypocrisy ?

    To marketers who are cheering this, I would advice to see Dr Pete’s presentation video in 2012 Mozcon. It certainly is an eye opener and I think Mozcast is one of the best new SEO tools out there.

    • re: Google Premium…

      have you ever noticed.. old GA numbers you did for a client a year ago are not the same?

      because as Google “improves” GA and reporting, they retroactively change the database.. they have to… so numbers you gave the client a year ago from their GA account, may or MAY NOT reflect the same numbers.. today..

      since it’s free, that is ok.. the moment they are not, Google becomes liable and accountable… that’s not in their business models.. #justsayin

  12. Pingback: Raven’s CEO on Ahrefs and choosing to end use of scraped data | Raven Internet Marketing Tools

  13. DeeSEO says:

    If rankings are flawed or don’t matter, why is Google trying to hide the data? #GoogleGroupies

    There’s all kinds of useful and valid SEO things you can do with ranking data. Advanced SEO’s know what I mean.

  14. Vic Drover says:

    I asked Rand Fishkin if this affected SEO Moz and he responded simply on Twitter that they were not pursuing adwords data or API.

    http://twitter.com/randfish/statuses/278013756003479552

    So SERP tracking by Raven competitors will continue until Google gets even more strict with their data.

    In addition, SERP placement is such an easy thing to explain to customers. Even if it became irrelevant tomorrow, folks will still desire this for many years, much like some folks are still adding meta keywords. Traditions are hard to break, especially for those not keeping current.

  15. Rank Tracking data passively and actively is important as well.

    Rankings in most of the tools aren’t always accurate. Moz is fixing some of that now.

    I’ve never depended rankings alone as a measurement of success. If there is no traffic to correlate what’s the point.

  16. Sure, SEO ranking is what we SEO’s bring to our client upfront as measure of success in doing online marketing. But, does our client really convert when they got in the first page of SERP? SEO ranking is just part of the bigger parcel of internet marketing, search engine optimization to be exact. There are other essential data that we need to focus on to better serve our client. Yes, tracking keyword rank eats up a lot of our precious time but I think, our clients deserve that extra effort. We just need to find new ways to still feed our clients with keyword rank and at the same time, feed them with more valuable information other than keyword ranking. I bet, the decision to scrape off the ranking tool has been a hard one for RavenTools and there are more valuable reason why they decide on scraping it off. May all of us prosper and progress with our businesses and cheers to the upcoming SEO challenges that we’ll surpass.

  17. Scott says:

    In comments to one of the Raven articles, someone pointed this out:
    http://www.mercurynews.com/business/ci_22169938/google-nears-deal-ftc-patents-portion-antitrust-dispute

    “… and preventing the export of data on advertising effectiveness to non-Google software so ad campaigns can be evaluated.
    The European Commission is investigating many of the same allegations.”

    If this includes complaints about the AdWords API rules, then others have been complaining about it so this is nothing new… although the article is about some kind of rumored agreement. We also don’t know if this legal case had any effect upon the AdWords staff, because there may be no need for the legal staff to guide AdWords management until the case has been resolved.

  18. Bill Bean says:

    Nice angle on this story, Scott.

    Search engine rankings = visibility. Rankings still matter. A simplistic approach to rankings is what must be abandoned.

    Businesses like Raven make difficult decisions all the time. What I do commend them for is leveraging the opportunity to get a lot of attention. Tasty lemonade made from Google lemons.

  19. White says:

    maybe you should create substitutes list, there’s a lot of other software with SERP tracking

    SheerSEO.com
    SEOmoz.com
    authoritylabs.com
    ….

    • Scott Offord says:

      I was hoping to publish a follow up post on alternative SERP tracking tools out there. This is a good start. Thanks.

      • Scott, thanks for this great article. I have been doing SEO for a very long time and every year I hear, “Rankings are dead” I would love to know how SEO’s who do not use rankings who are heralding this as a good thing do SEO and even more importantly I would love to see the results they get. Even if you do not send the ranking reports to the client and only measure against the bottom line traffic and conversions how do you make that traffic and conversions happen. Take for example, a keyword with high volume very targeted to the client’s business in the #15 position. Move it to #3 and now you get your bottom line but how do you do that if you are not tracking what position it is in? Also, what happens when a major money keyword drops out of the top 1o due to a Google update, if you do not know that how do you know to ramp up your Adwords position for that keyword until you work out what needs changing. Any SEO who says they do not use ranking data does’nt do advanced SEO. We use SyCara.com for ranking data my favorite part is ranking historical rankings side by side with current. Taking groups of keywords that I am working on and tracking their progress to traffic and conversions.

      • White says:

        welcome :) , it’s a pretty lame list, I think there’s dozens of tools/software

    • Rebecca says:

      Hey White!
      You can add RankRanger.com to that list – they built their own SERP tracker and are independent of Google Adwords API.

  20. Zarocni prstani says:

    Hmm .. thanks for youre inputs folks ! So we cant find a good source beside Majestic ? Situation when we are depedant on single player (Majestic in this case … not BigG) is never goog …. Anyone can confirm that link-assistant-com has his private spider as they say ?

  21. Alexandra - Link-Assistant.Com says:

    Thanks for your post, Scott!

    We ran a survey among our software users in June – November of 2012.
    It actually collected a number of questions, including one with the options. We asked “What’s the main problem you’re trying to solve with the help of our SEO software?” The options were (the wording might be a bit different): 1) My site needs more visitors 2) I want my website to generate more profit 3) I want to rank higher in search engines for my keywords 4) I want to spend less time on SEO 5)Other
    The surprise came when it turned out over 80% selected option 3 “rankings” EVEN though they had hints about other possible reasons to do SEO (with software, in our case).

    So I’m not surprised clients still want to see rankings. It somehow goes past most minds that rankings themselves are not the goal. And even though professional SEOs support this opinion, clients choose rankings as the most tangible proof of agencies’ work. And, while gently trying to educate their clients, agencies will still have to get these ranking reports to them. Things won’t change just over a few months.

    Especially now when lost of other software houses (including us at Link-Assistant.Com) continue providing ranking data.

  22. Michael says:

    Great post! Personally, i think that SERP tracking is very important thing. Google should build own tool if they don’t want scrappers. Unfortunately, their Webmaster Tools are not excellent now. If it’s ok i would recommend tool which i’m actually using – ColibriTool – great young and promising tool and it’s not only a serp tracker.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>