XML Sitemap Question!
-
Hi All,
I know that the sitemaps.xml URL must be findable but what about the sitemaps/pageinstructions.xml URL?
Can we safely noindex the sitemaps/pageinstructions.xml URL?
Thanks!
Yael
-
In regards to an "XML site map"- it's essential to have a site-map on your company website, that's because it makes it much easier for a Google Bot, to find the pages/ blog posts to crawl and index these if you have a XML site map.
We added a Yoast SEO site map to our Bristol Garden Room company website, in Bristol to ensure that all the blog posts / main pages were getting indexed.
-
Yes, you can noindex practically anything using the robots.txt file
-
Hey
Yes, you can safely do that. In the end, you mostly want to make sure that the right pages are being Crawled > Indexed. If that requires certain (XML) files to be no-indexed, then that's the way you want to go.
Martijn.
-
Hey Yael,
You should be able to. If it's being indexed, you can request removal via Bing Webmaster Tools and Google Search Console. You can also update your robots.txt file to disallow the page "http://www.robotstxt.org/robotstxt.html"
I hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Only a fraction of the sitemap get indexed
I have a large international website. The content is subdivided in 80 countries, with largely the same content all in English. The URL structure is: https://www.baumewatches.com/XX/page (where XX is the country code)
Intermediate & Advanced SEO | | Lvet
Language annotations hreflang seem to be set up properly In the Google Search Console I registered: https://www.baumewatches.com the 80 instances of https://www.baumewatches.com/XX in order to geo target the directories for each country I have declared a single global sitemap for https://www.baumewatches.com (https://www.baumewatches.com/sitemap_index.xml structured in a hierarchical way) The problem is that the site has been online already for more than 8 months and only 15% of the sitemap URLs have been indexed, with no signs of new indexations in the last 3 months. I cannot think about a solution for this.0 -
Migration developer question
Hi Guys, We are in the process of migrating our website and are moving to: AWS/Elastic Beanstalk hosting and the only way to do a custom domain with a third-party (not Amazon) DNS Service is by setting up a CNAME that points to the EBS Instance. Do you think this will impact SEO performance in any way? Cheers.
Intermediate & Advanced SEO | | cerednicenko0 -
Can't generate a sitemap with all my pages
I am trying to generate a site map for my site nationalcurrencyvalues.com but all the tools I have tried don't get all my 70000 html pages... I have found that the one at check-domains.com crawls all my pages but when it writes the xml file most of them are gone... seemingly randomly. I have used this same site before and it worked without a problem. Can anyone help me understand why this is or point me to a utility that will map all of the pages? Kindly, Greg
Intermediate & Advanced SEO | | Banknotes0 -
Content Audit Questions
Hi Mozzers Having worked on my companies site for a couple of months now correcting many issues, im now ready to begin looking at a content review, many areas of the site contain duplicate content, the main causes being 1. Category Page Duplications
Intermediate & Advanced SEO | | ATP
e.g.
Widget Page Contains ("Blue Widget Extract")
Widget Page Contains ("Red Widget Extract")
Blue Widget Page Contains ("Same Blue Widget Extract")
Red Widget Page Contains ("Same Red Widget Extract") 2. Product Descriptions
Item 1 (Identical to item 2 with the exception of a few words and technical specs)
Item 2 Causing almost all the content on the site to get devalued. Whilst i've cleared all moz errors and warnings im certain this is causing devaluation of most of the website. I was hoping you could answer these questions so I know what to expect once i have made the changes. Will the pages that had duplicate content recover once they possess unique content or should i expect a hard and slow climb back? The website has never receive any warnings from Google, does this mean recovery for penalties like duplicate content will be quicker Several pages rank on page 1 for fairly competitive keywords despite having duplicate content and keyword spammy content. What are the chances of shooting myself in the foot by editing this content? I know I will have to wait for google to crawl the pages before i see any reflection in the changes, but how long after google has crawled the page should I get a realistic idea of how positive the changes were? As always, thanks for you time!0 -
Sitemap on a Subdomain
Hi, For various reasons I placed my sitemaps on a subdomain where I keep images and other large files (static.example.com). I then submitted this to Google as a separate site in Webmaster tools. Is this a problem? All of the URLs are for the actual site (www.example.com), the only issue on my end is not being able to look at it all at the same time. But I'm wondering if this would cause any problems on Google's end.
Intermediate & Advanced SEO | | enotes0 -
Rel="prev" and view all question
Okay, I've read the posts by Google about the new prev, next tags and the suggestion for using a view all option. I've also read the posts here on SEOMoz on the topic but none of them quite address what we have. First, Some of our main categories are very large (over 6000 pieces of jewelry) so a view all option would take forever to load be completely useless to a visitor. Second, our category home pages provide (here's an example😞 A description of the category with links to important sections and articles A row of new items A dozen of the popular items from the category. Links to related articles if applicable. So we have a real category home page with content instead of just categories that start immediately with pages of product. Should we set the canonical url for all of the browse pages to the main category page, create a view all page or just use the next and previous rel tags with the category home pages as the first in the series?
Intermediate & Advanced SEO | | IanTheScot0 -
Advanced Question on Synonym Variation Pages!
Hi, This is quite an advanced question, so I'll go through in detail - please bare with me! I launched the new version of our website exactly a week ago - and all the key metrics are in the right direction: Pages / Visit +5% , Time on Site +25%, Bounce rate down 1 %. I work in an industry were our primary keyword has 4 synonyms and our long tail keywords are location related. So as an example I have primary synonyms like: Holiday, Vacation, Break, Trip (Not actually these but they are good enough as an example). Pluralised versions and you have 8 in total. So my longtail keywords are like: Las Vegas Vacation / Las Vegas Vacations
Intermediate & Advanced SEO | | James77
Las Vegas Holiday / Las Vegas Holidays
Las Vegas Trip / Las Vegas Trips
Las Vegas Breaks / Las vegas Breaks All these synonyms effectively mean the same thing, so my thinking on my new website was to specifically target each of these synonyms with their own unique page and optimise the meta and page titles, to those exact words. To make these pages truely unique, I therefore got a bunch of copywriters to write about 600 words unique for every long tail synonym (well over 750,000 words in total!). So now at this point I have my page "Las Vegas Holidays" with 600 unique words of content, and "Las Vegas Vactions" with 600 words of unique content etc etc etc. The problem is, when the user is searching for these words, there primary goal is not to read 600 words of content on "Las Vegas Holidays" - their primary goal is to get a list of last vegas holidays that they can search, view purchase (they may want to read 600 words of content, but is not their primary goal). So this puts me in a dilema - I need to display the nuts and bolt (IE the actual holidays in las vegas) to the customer on any page they land on off my synonyms as the primary content. But to make sure these pages are unique I need to also have this unique content on that page. So here's what I did: On every synonym version of the page I display the exact same information. However, on each page I have a "Information" link. and on click this pop's up a layer which contains my unique content for that page. To further optimise using perfect anchors in this content pop-up, I have cross linked the synonym pages (totally naturally) - IE on my "Las Vegas Holidays" page, in the content I may have the words "Las Vegas Breaks" - this would be linked the the "Las Vegas Breaks" synonym page. In theory I don't think there is anything wrong with what I am doing in the eyes of the customer - but I have a big concern that this may well look "fishy" to SE's. IE the pages are almost identical to the user except for this information pop-up layer of unique content, titles and meta. We know that Google at least can get can tell exactly what the user see's when they land on that page ( from their "Preview") and can distinguise between user visible and hidden text. Therefore, even though from a user experience, I think we are making a page that is perfect for them (they get the list of vactions etc as the primary content, and can read infomation if they want by clicking a button), I am concerned that SE's are going to say - hold on a minute there are load of pages here that are identical except for a chuck of text that is not visible to the user (Even though this is visible to the user if they click the "Information" button), and this content cross links to a load of almost identical pages with the same thing. Today I checked our rankings, and we have taken a fair whack from google - I'm not overly concerned at the moment as I expected big fluctuations from ranking for the first few weeks - but I'd be a lot more confident if they were fluctuating in the right direction!! So what do I do?
As far as I can see my options break down as follows: Content Display:
1/. Keep it as it is, and hope the SE's don't see it as spammy. Even though I think what we are doing is the best for customer experience, I'm concerned SE's won't. 2/. On every synonym page, below all the list of products, packages etc that the customer wants to see, display the unique content as a block of subtext text which is visble by default. This however could make the page a bit ugly. 3/. Display a visible snippet of the unique content, below all the packages, and have a more button which expands the rest of the content - IE have a part visible layer. This is slightly better for display, but again I'm only displaying a portion of visible content and the rest will still be flagged as "hidden" by default to the SE's. Cross Linking within the content:
1/. Keep it as it is where synonym keywords link to the synonym version of the page. 2/. Alter it so that every sysnonym keyword links to the "primary" synonym version of the page - EG if I now "Las Vegas Holidays" is my main keyword, then "Las Vegas Vactions" keyword, would not link to my "Las Vegas Vactions" page as current, but would link to my "Las Vegas Holidays" page. I apologise for the indepth questions, but it requires a lot of explanation to get it across clearly. I would be grateful on any of your thoughts. Many thanks in advance.0