Search Engine,Optimization,Services

Keywords:

Keywords are the words that are used to reveal the internal structure of an author's reasoning. While they are used primarily for rhetoric, they are also used in a strictly grammatical sense for structural composition, reasoning, and comprehension. Indeed, they are an essential part of any language.

There are many different types of keyword categories including: Conclusion, Continuation, Contrast, Emphasis, Evidence, Illustration and Sequence. Each category serves its own function, as do the keywords inside of a given category.

Website and Search Engine Keywords:

On the web, a keyword is a reference to the content and/or the type of meta element included in a given web page's HTML code to aid in the page's indexing. A keyword meta element may include several comma-separated keywords (or keyword phrases, each of which may contain several individual words) as follows [1]:
<meta name="keywords" content="keyword,another keyword,one more keyword" />

Looking back in the 90's, the search engine crawlers were relatively poor in terms of analysis capabilities, and, thus, the meta keywords were a simple way to detect the topics of a page. Historically, it has been the first way to optimize for search engines, however, no major search engine today claims to read the keywords which has raised the question of whether they are still needed at all.

Query terms:

The term "keyword" also refers to the terms or phrases submitted by a user of a search engine. For example, a search of the phrase "keyword search" via the Google search engine reveals a set of search engine results that relate to the specified topic "keyword search". The link with the meta keywords previously defined was real in the last century, however, the search engine are actually using much more advanced techniques (statistics, natural language processing, web topology...) to enhance their results and thus the decreasing relation between those two type of keywords.

But a technical definition of "keyword" does not provide insight to their significance or how to work with them successfully. Significance is straightforward: as more and more of human knowledge is digitized and therefore 'searchable,' the ability to understand and successfully develop, organize and manipulate keywords leads directly to access to critically important information, and gets information to the right audience.

Working with keywords:

There are two aspects to working with keywords: from the perspective of the information provider, and from the perspective of the information users (i.e. "producers" and "users").

Information producers: For information publishers (meaning anyone providing digitally searchable content they want to be found by a given audience), there are a tremendous number of subtleties to developing a keyword framework that a) adequately describes the content, and b) connects that content to the right audience. The first mistake many publishers make is to 'underdescribe' their content by using a keyword that is too general to be useful. For example, to say the keyword for this essay is "keywords" would be such an 'underdescription' -- a better keyword (really, keyword phrase) would be something like "keyword development" or "keyword definition" or "how keywords are used." The second mistake publishers frequently make is to not put themselves in the mind of the searcher, but to instead use keywords that are relevant to them. The easy fix for this lack of perspective is simply to do the footwork: make a list of keywords that might be relevant and then verify whether or not they garner searches by checking the list on a database that collects such information and provides "suggestions" that you may never thought of (Keyword Discovery, Wordtracker and Overture are just three such services). Never make assumptions -- for example, according to one of the keyword databases listed, "keyword assistance" gets zero searches per year but "keyword research" gets 50 searches per year. [Note that a very common error is the "verify" a keyword by typing it into a search engine and seeing how many web pages come back. This indicates is how many pages have that keyword in their content, not how many people are searching for that keyword, and there is no relationship between those two datum.] It is important to keep in mind the need to be flexible -- there are as many ways to describe something (and develop a search query for it) as there are people with keyboards -- but not too flexible. The goal is precision, and the searcher will appreciate efforts to describe precisely what your content is about if it is precisely what they are looking for.

Information users: Precision of the keyword phrase is of paramount importance to the searcher. Search engines are so powerful that they frequently return listings that ranges from exactly the user intent to completely irrelevant results. Careful consideration of exactly what the searcher wants is a prerequisite. Even more, searchers need to be familiar with ways to structure a search to get the information they want.

It is well worthwhile to investigate the advice search engines provide for successful querying. Some of the very useful tools for structuring keyword phrases include quotes, brackets, and boolean operators:
Quotes: Placing a keyword phrase in quotes tells the search engine "Return pages/documents with these words, in any order"
Brackets: "Return pages/documents with these words in exactly this order"
AND: AND is used to logically connect two search queries. [Note -- AND must be capitalized in the search query] "Find an instance of these two keyword phrases within 25 words of each other." Very handy for “localizing” a search, for example: “Ethiopian restaurant AND St. Louis”.
NOT: NOT is used to exclude pages/documents that contain two identified keyword phrases. [Note -- NOT must be capitalized in the search query.] For example, if the searcher wanted information about Paris the City of Light, not Paris the celebrity, the proper query would be "Paris NOT Hilton." Some search engines uses "-" (minus sign) instead of NOT; had this been the case the search would have been "Paris - Hilton".

Note there are no universal syntax conventions. Different search engines implement different grammar rules for the above-mentioned productions.

Indexing :

Many of the most popular and comprehensive search engines on the Web are indexing engines - also known as crawlers or spiders. They get these names due to their particular way of finding information on the Internet - they have a program (often referred to as a "bot") that scans or "crawls" through Web pages, classifying and indexing them based on a set of pre-determined criteria. The weight given to these criteria, which may include links to your page from other sites, keywords, their positioning on a page and meta-tags, depends upon the individual indexing engine, and makes up their ranking algorithm. The information gathered during the crawling process is placed into a database, called an "index", which is then searched every time you enter a keyword query at their site. When you perform a search at an indexing engine, then, you're not actually querying the entire Web, but the portion that they have examined and included in their database.

Indexing search engines are best to use for hard to find information or very specific data, as they search through a wide and varied database of sites, returning many results. If your query is too broad, however, you risk getting an overwhelming amount of results (numbering in the hundreds of thousands or more!

Directories

Directories are categorized groupings of sites, most often compiled and organized by human editors. They're organized into a series of categories and sub-categories, moving from the general to the specific. Each sub-category brings you to a list of additional sub-categories, until finally you reach a list of sites. While the quantity of results are usually much fewer than those returned by an indexing engine, their relevancy and quality are usually much higher.

For ease of use, most directories also have a search feature, which enables you to search through their listings - a word of caution, however: these search functions only search through the directories' categories and listings (i.e. titles, descriptions and URLs as they appear in their database) and not the sites themselves.

Directories are great to use when you don't know a lot about a subject, need help narrowing down a topic, or when you're looking for general information.

Natural Language :

If you're a beginner to the Internet, or prefer to "ask" your questions (for example, "Why is the sky blue?" "What is the temperature of the sun?" etc.), rather than trying to formulate a keyword query, then a Natural-Language search engine is the way to go. These allow queries to be submitted in the form of a question, and then help you to narrow down your search by clarifying what it is you're looking for. Sometimes, they'll even provide the answer to your question directly on the search results page!

"Pay" engines :

With the increasing popularity of search engine advertising, paid inclusion and pay-for-placement services abound, and are offered by most major search engines. In a nutshell, these programs require payment in order to have your site listed with them.

Paid Inclusion
Paid inclusion services require a fee in order to list a site in their database. It can take the form of a yearly fee for a directory listing, or can be a cost-per-click listing in an index - where the site owner pays every time someone clicks on their link. It could also be a combination of a flat fee and/or cost-per-click payment method (just to make things confusing!).

The most important thing to know about paid inclusion, however, is that placement or ranking within the search engines' results set is not guaranteed - i.e. a site may be included, but it will not receive preferential treatment. Some search engines that have paid inclusion programs still offer a free (slower) submission process, though these are sometimes reserved for non-commercial sites.

Examples of search engines with paid inclusion programs are: Yahoo and Entireweb.

Pay-For-Placement
Pay-for-placement (or pay-per-click, cost-per-click) programs usually take the form of an auction-style environment in which site owners try to outbid each other to get their sites listed higher up in the results. Payment is in the form of a CPC (Cost-Per-Click) whereby the site owner pays a certain amount every time someone clicks on their link.

As pay-for-placement programs are more like advertising than search results, pay-for-placement engines no longer try to attract users to their own sites, but rather distribute their paid results to other search engines, to be displayed as "Sponsored Listings" above or alongside regular results.

Pay-for-placement results are best for when you're searching for something to purchase. The vast majority of listings are for retail sites or online services that are willing to pay for potential customers.

Metasearch :

Every time you type in a query at a metasearch engine, they search a series of other search sites at the same time, compile their results, and display them either by search engine employed or by integrating them in a uniform manner, eliminating duplicates, and resorting them according to relevance. It's like using multiple search engines, all at the same time.

By using a metasearch engine, you get a snapshot of the top results from a variety of search engines (including a variety of types of search engines), providing you with a good idea of what kind of information is available.

Meta-search engines are tolerant of imprecise search terms or inexact use of operators, and tend to return fewer results, but with a greater degree of relevance. They're best to use when you've got a general search, and don't know where to start - by providing you results from a series of sites, they help you to determine where to continue focusing your efforts (if this proves necessary). They also allow you to compare what kinds of results are available on different engine types (indexes, directories, pay-for-placement, etc), or to verify that you haven't missed a great resource provided by another site, other than your favorite search engine (acting as a backup). Overall, they're a great way to save time.

A Standard for Robot Exclusion
Table of contents:

This document represents a consensus on 30 June 1994 on the robots mailing list (robots-request@nexor.co.uk), between the majority of robot authors and other people with an interest in robots. It has also been open for discussion on the Technical World Wide Web mailing list (www-talk@info.cern.ch). This document is based on a previous working draft under the same title.

It is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots.

The latest version of this document can be found on http://www.robotstxt.org/wc/robots.html.
Introduction
WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. For more information see the robots page.

In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting).

These incidents indicated the need for established mechanisms for WWW servers to indicate to robots which parts of their server should not be accessed. This standard addresses this need with an operational solution.
The Method
The method used to exclude robots from a server is to create a file on the server which specifies an access policy for robots. This file must be accessible via HTTP on the local URL "/robots.txt". The contents of this file are specified below.

This approach was chosen because it can be easily implemented on any existing WWW server, and a robot can find the access policy with only a single document retrieval.

A possible drawback of this single-file approach is that only a server administrator can maintain such a list, not the individual document maintainers on the server. This can be resolved by a local process to construct the single file from a number of others, but if, or how, this is done is outside of the scope of this document.

The choice of the URL was motivated by several criteria:
The filename should fit in file naming restrictions of all common operating systems.
The filename extension should not require extra server configuration.
The filename should indicate the purpose of the file and be easy to remember.
The likelihood of a clash with existing files should be minimal.
The Format
The format and semantics of the "/robots.txt" file are as follows:

The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form "<field>:<optionalspace><value><optionalspace>". The field name is case insensitive.

Comments can be included in file using UNIX bourne shell conventions: the '#' character is used to indicate that preceding space (if any) and the remainder of the line up to the line termination is discarded. Lines containing only a comment are discarded completely, and therefore do not indicate a record boundary.

The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below. Unrecognised headers are ignored.
User-agent
The value of this field is the name of the robot the record is describing access policy for.

If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record.

The robot should be liberal in interpreting this field. A case insensitive substring match of the name without version information is recommended.

If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.
Disallow
The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html.

Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record.
The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.
Examples
The following example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/" or "/tmp/", or /foo.html:
# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
This example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/", except the robot called "cybermapper":
# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space

# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
This example indicates that no robots should visit this site further:
# go away
User-agent: *
Disallow: /
Example Code
Although it is not part of this specification, some example code in Perl is available in norobots.pl. It is a bit more flexible in its parsing than this document specificies, and is provided as-is, without warranty.

Removing single URL from the Google Index:

In certain cases, on course to the lifting of penalties ( solving usability or content related problems, removing unwanted content, requesting re-evaluation ) removing an offending web page / access to a faulty URL may become necessary. Such changes will take effect in a web site's ranking during subsequent crawls and updates, or in case of a ban, will be viewed as the proper action taken when re-evaluating the content. However, entirely removing a URL from the index is rarely necessary, as the historic supplemental results and cached versions of a URL ( that already return server responses for deleted or permanently redirected pages ) are not determinative of the outcome.

For more generic purposes, Google will remove any and all URLs from its index if its crawl requests consistently receive a 404 - not found or 410 - Gone HTTP status code message from the server. Using a permanent redirect ( status code 301 ) will gradually transfer all parameters of the source URL to the target of the redirect, and once the transition is complete the source page may be dropped from the Index. URLs to obsolete pages are likely to first become "historic" Supplemental results, then fade out from the index completely. For the removal of valid URLs that are otherwise accessible, using the robots.txt disallow features and/or limiting Googlebot indexing with META tags are the proper methods.

Known issues:

Sometimes web site owners may feel that a page they are administering, and is currently in the Google index, should no longer be listed as a result. Removing the page from the server, and thus the server responding to the requests of Googlebot with a 404-not found server message ( 404 - not found / 410 - gone HTTP status code ) will eventually mark the given web page as supplemental, and not show it as a result for normal searches. Such historic supplemental pages may stay within the index, and may be reached by queries unique to the deleted page up to a year. During this time a copy of the last crawled version will remain in the Google cache. Such historic supplemental pages will not play a role in evaluating a web site for relevance or importance, and may generally be ignored completely by the algorithms. Unless there are legal issues with the now deleted, but still cached content of the web page, copyright breach, defamation or security problems with the information displayed, you should not need or care to remove the page entirely from the index. In these cases you may request the cache to be deleted through the Google webmaster help center, using the tool for removing URLs. Regarding the manual URL removing system, it only works if the request corresponds with either a 404 - not found server response, or a Googlebot related restriction on the pages and in the robots.txt file. Also note that pages excluded are still cached and indexed, and only remain hidden from the users for an estimated 180+ days period ( exactly 6 months or more ). The URL removal tool thus can not be used to clear the history of a web site, rather to exclude it from the search result pages. Its combination with the Reinclusion request for dealing with penalties and bans is thus redundant and generally advised against.

+ Resolution: Completely removing an otherwise valid URL from the Google index, or preparing to remove a soon to be deleted page should be done by restricting the crawling, indexing, caching and display of the pages on a case by case basis. Implementing the proper META tags into the HEAD section of a page will communicate the necessary information to the algorithms, and subsequent crawls and updates will see the page gradually be excluded from the results, and clear the associated cache as well. Using robots.txt disallow features will be reported as a temporary block against the crawling of these URLs, and will only work while it is in place. It is advised to first include the page specific directives, then once these have been recognized, the removal of the pages or the setup of the robots.txt disallow attributes can follow. Also it is important to note that if a URL is constantly referred to by links from other web sites, it will be tried against the robot directives over and over again, and once those are not restricting the crawl, and the page at the URL is still in place, it will be re-indexed. After about a year of a URL becoming unavailable for Googlebot to crawl, it's usually dropped from the index entirely, including the cache and occasional supplemental versions. <META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOARCHIVE, NOSNIPPETS"> is to be placed in the HEAD section of the pages that are to be excluded from the index. Another method is the <META NAME="GOOGLEBOT" CONTENT="NOFOLLOW"> directive that instructs Googlebot not to crawl the URLs that the page on which this directive is found links to. Use with caution. Below is an example for a robots.txt entry to block Googlebot from crawling certain URLs. You may place the robots.txt file into the relative root directory of your web site, or the directory in which it needs to applied. You can enter a list of URLs relative to the path, or use but the the / character to extend the directive to the entire directory ( if the robots.txt is in the domain root, this will put all the URLs of the domain or subdomain on the list of exceptions ), or use directory names with / at their end to exclude them. You may also speed up the process of removing a URL from the Index by first restricting their crawl and / or deleting them, and request the URL to be removed through the URL removal request page of Google Webmaster Tools. An example on robots.txt for disallowing an entire directory ( if placed in the domain root, i.e.. is reachable through www.example.com/robots.txt , this directive will disallow the crawl of the entire domain ) :

# Disallow Googlebot
user-agents: googlebot
disallow: /

Another example disallowing a directory, using a path relative to the position of the robots.txt file:

# Disallow Googlebot
user-agents: googlebot
disallow: directory/

Another example disallowing specific files, using a path relative to the position of the robots.txt file :

# Disallow Googlebot
user-agents: googlebot
disallow: directory/file1.html
disallow: file2.html
disallow: anotherfile.php

Meta Tag Resources:

As a search engine prowls your site, it gathers information from the title, headings, content, and Meta Tags such as description or keywords. It compares the words within each of these sections and "ranks" the site dependent upon how well the information matches. We have more information on how to maximize your meta tag references below.

It is important for website developers to understand that a default installation of WordPress does not contain the description and keywords meta tag data. Meta tags can be added manually, through changes to the Theme template files or through WordPress Plugins.
What Are Meta Tags?

The word meta means information about. Meta Tags were created early on to provide concise information about a website. Meta tags list information about the web page, such as the author, keywords, description, type of document, copyright, and other core information.

This is an example of a meta tag for description:
<meta name="description" content="This is the
description sentence or short paragraph about
the article or post." />

The most common meta tags examples include:
<meta name="resource-type" content="document" />
<meta http-equiv="content-type" content="text/html; charset=US-ASCII" />
<meta http-equiv="content-language" content="en-us" />
<meta http-equiv="author" content="Harriet Smith" />
<meta http-equiv="contact" content="harrietsmith@harrietsmith.us" />
<meta name="copyright" content="Copyright (c)1997-2004
Harriet Smith. All Rights Reserved." />
<meta name="description" content="Story about my dog
giving birth to puppies." />
<meta name="keywords" content="stories, tales, harriet, smith,
harriet smith, storytelling, day, life, dog, birth, puppies, happy" />
Why Are Meta Tags Missing?

In the default installation, WordPress does not include meta tags such as description and keywords. Why? Well, let's look at the above tags.

The second and third tags set the language and character set for the page. This example sets the character set to be in the English language style as found in the United States, using the ASCII character set. This means that the page will probably feature spellings like "center" instead of "centre" and "humor" not "humour". It also gives information to the browser to recognize the characters as not being Chinese.

The author and contact information lists a specific person. The description tag lists a description of the post that is unique to that post. The keywords also list words found within that post. Are you seeing the pattern?

All of this is unique information. WordPress may do some magical things, but it can't read your mind. If you want to supply search engines better information that is more specific to your web pages, you have to add the meta tag data yourself.
Are Meta Tags Necessary?

A good question to ask is if meta tags are still necessary. They used to be more helpful, providing important information to the Internet browser. As browsers became more sophisticated, they stopped needing a lot of hand holding in order to figure out if your site is in English or Chinese.

Some search engines don't use the meta tag information any more because many people abused it. In fact, meta tags may not represent the content of your site. But, it still doesn't hurt your status with search engines if you make use of these little bits of information, and do so truthfully.

Without a doubt, content is the biggest contributor to search engine page ranking, so if you want to raise your rankings, make sure you have quality content.

Keywords meta tags, on the other hand, are still used by some search engines to categorize and rank your website. Those engines compare the keywords with the content, giving you "points" for having your keywords match your content. Keywords are one of the most important meta tags you can add to your WordPress site.

Covering all the reasons meta tags are and aren't important to search engines is beyond the scope of this article. In the Resource section below are some links to more information on meta tags and their impact on search engines.
Putting Meta Tags Back In

To add meta tags to your site, simply add them to the header.php template file in your WordPress Theme, specifically in the head section near the link for the style sheet. At the top you will see the DOCTYPE tag and below that you will see a couple more tags and then the <title> tag, looking something like this:
<title /><?php bloginfo('name'); ? /><?php wp_title(); ? /></title />

Below this line you can add your meta tags. You can add meta tag information such as the content, language, author, contact, and copyright, since these are basically the same on every page of your site.

But what about the 'dynamic' types of information such as description and keywords? This information is unique to every web page on your site. Putting them in the header.php means the information won't change throughout your site.

What you need is something to dynamically add keywords and descriptions on a per-post basis. To add a description, keywords, and other meta tags that are unique to each post or Page generated, you have two choices: you can add them as generic references or you can use plugins.
Generic Meta Tags

If you have a clear purpose for your website, then you can use that information to create generic meta tags and place these in your header. Let's say Harriet Smith is a veterinarian, and enjoys sharing her animal stories on her WordPress blog.

A description meta tag that would describe all of her posts might be:
<meta name="description" content="Special stories and
tales about dogs, cats, birds, ferrets, and other
domestic animals as told by Harriet Smith,
veterinarian. Stories include tales of animal
bravery and courage, life and death, companionship,
and the wonderful joy animals and pets bring to
their human partners." />

If Harriet writes a story about a dog giving birth in the middle of a hurricane, and accompanying the dog were two humans who risked their lives to stay with the dog during the delivery, the descriptive words of bravery and courage, life and death, pets and human, would surely apply. Another story about an animal that returns home to its owner after missing for 5 years might also be described with the words of bravery, courage, life, death, and companionship.

From these two stories, Harriet could create a good generic set of keywords:
<meta name="keywords" content="stories, tales,
harriet, smith, harriet smith, storytelling, veterinarian,
vets, animal doctor, bravery, brave, courage, life, lives,
death, dying, pets, human, companionship, dog, cat, birds,
ferrets, pets, pet, birth, puppies" />

With these generic description and keywords meta tags, Harriet's website would be accurately described and that should please just about any search engine.

This is the manual style of adding a description, but WordPress can make this easier for you. Add your description, similar to above, in your Admin > Options > General panel. The description is entered in the line designated as Tagline. Then paste in the following in your head area.
<meta name="description" content="<?php bloginfo('description'); ?>" />

WordPress will automatically generate the description. Note: Some Themes use this tag in the header just below the title of the blog or site. If you don't want it there, remove or comment out the tag in the header area.

To get more sophisticated, you can add a conditional tag query that asks "If this is a single post view, show the post title; if this is a multi-post view, show the blog name and description." The conditional tag query looks like this:
<meta name="description" content="<?php if ( is_single() ) {
single_post_title('', true);
} else {
bloginfo('name'); echo " - "; bloginfo('description');
}
?>" />
Using Meta Tag Plugins

There are several Plugins that allow the blog administrator to set the keywords, description, and other meta tags to be unique on each post. These plugins make use of the Custom Fields in the Write Post Panel. You can find meta tag Plugins in the Official WordPress Plugin Directory.

Using Meta Tag Plugins allows you to customize each of your meta tags, and to choose the ones you want to add, or not, on a per-post basis. This allows your meta information to better describe each page.
Meta Tag Resources

Here are some sites that will help you learn more about meta tags, how they work, and why you should or shouldn't use them.
Search Engine Watch
UC Berkeley's Guide to Search Engines
Search Engine Watch's How to Use HTML Meta Tags
WebDeveloper's Back to Basics: Meta Tags
W3 Organization's Global Structure of an HTML Document and Meta Tags
header.php code and tutorial outputs the correct robots, googlebot, and msnbot meta depending on page, single, home, archive.

More information to WEBMASTERS for GOOGLE applications.

GOOGLE Section One click here

GOOGLE Section Two click here

GOOGLE Section Three click here