The Meta Keywords and spider-oriented ""Hello Spider"" text
Ok, you've got your beautiful Website and a beautiful set of Meta keywords and Meta
Descriptions, haven't you? And you've created a robots.txt file in the root of your
site, I presume... If not you REALLY should go back and search for some META
information and come back as soon as you understand what we are talking about...
Ok, I trust you and if you're here the part over is OK. Now your site is ranked
10th at best? I DO NOT have an answer. But I can give you some useful tips in ASP
for making a better ranking.
Searching for more info?
If you're a really cool guy who wants to find out more info on rankings, spiders and
so on, you can take a look at Google Webmaster's Guide, for example. More inforamtion
about Robots can be found in the extremly exaustive RobotsTxt.Org Guide.
Finally, to submit a page to Google take a look here.
The Basics
First of all, inspite of the title, we are not cheating. We are simplifying the work
of the search engine by helping its "little workers" to do their work better
and faster. You must know, before proceeding, that web search engines do their work
using spiders or better web-spiders. A web spider is a sort of automatic browser that
does the work of connecting to a page, looking into it for the content and archiving
it, along with following links provided in it. So sometimes in the life of your site
you'll see (if you have statistics for your site) some strange browser type
(for example Java1.4.0_03, libwww-perl/5.69 or GoogleBot). This is the clear
symptom your website has a bad spider infection! ;)
A helping hand...
Sometimes the spider does its job in the correct way, sometimes it doesn't.
And so, for "forcing his hand" a little, many sites try to append at
the end of the page some content (for example "sex, sex, sex, sex..:"
one zillion times...) for elevating their ranking in that keyword. It is now
clear that a zillion repetitions often does the opposite of the intended
function of and makes the spider avoid that page, but it is also clear that having
a couple of "salient" words published is not only important, but
sometimes vital. Think about an "image-based" entry page for your site.
You'll not have the chance to give the spider the correct information about the
content of that peculiar page, and so it just drops down.
So, A good solution is to place some words in the page for the sake of spiders.
But spiders are growing much and much intelligent... And so, if some time ago
the "normal" practice was to write "white on white" or
"black on black", nowaday you must write in a meaningful color and
NOT into strange DIVS or hidden LAYERS. It is a little ugly to see a bunch
of no-meaning words in the middle of your website, and so many have dropped
this useful tip.
This is mine, that is yours...
Choosing and sorting is the core of the World... and even ASP has the capabilities
to "understand" who is coming. I've seen many examples on the web that
implement a sort of "redirect the spiders to another page" mechanisms.
It doesn't work. The spider will be aware of the redirect (302 error code: moved)
and will not index the SOURCE page, only (maybe) the destination. And that's bad.
In the following lines I'll guide you to implement a strange kind of page. A page
that behaves differently on "normal browsers" and on "spiders",
allowing the first to see the "stylish" page, and the latter to see a
"stylish" page with a "welcome spider" section.
The Code...
First of all let's open a file named "cheating.asp" which will contain only the functions.
It will look like this:
<%
' The spiderWelcomeText will write a "Hello Spider"
' text in the dedicated paragraph. Sort of
' making a fool of the spider, or better clarifying
' what these things are to accidental non spider
' folks with very strange browsers.
dim spiderWelcomeText
' The spiderMetaKeys is a comma-separated list of words
' the function will use to generate the text used for
' cheating the spider
dim spiderMetaKeys
' The spiderLinksText will write a friendly text
' before more important links to be added
dim spiderLinksText
' The spiderLinksHtml will contain the HTML code for the Links
dim spiderLinksHtml
Function isSpider()
' Well, normally the browser isn't a spider...
isSpider = 0
' No other meaning than forcing the isSpider behaviour
' for testing pourpose
if request("spider") = 1 then isSpider = 1
' Takes the name of the UserAgent currently used and put it
' into lower case for compairson
agent = lcase(Request.ServerVariables("HTTP_USER_AGENT"))
' Now, most of the Bots refers to themself as libwww,
' java, perl, crawl, bot. let's start with some conditions
' If the agent contains "bot" then it is a Spider
if instr(agent, "bot") > 0 then isSpider = 1
' If the agent contains "perl" then it is a Spider
if instr(agent, "perl") > 0 then isSpider = 1
' If the agent contains "java" then it is a Spider
if instr(agent, "java") > 0 then isSpider = 1
' If the agent contains "libw" then it is a Spider
if instr(agent, "libw") > 0 then isSpider = 1
' If the agent contains "crawl" then it is a Spider
if instr(agent, "crawl") > 0 then isSpider = 1
end function
function spiderContent()
' First of all we must analyze if the browser is a spider or not.
' It isn't useful to waste machine time playing with keywords
' if it isn't...
if isSpider() then
spiderContent = ucase(spiderWelcomeText) & "<br>" &_
lcase(spiderMetaKeys) &_
", " & ucase(spiderMetaKeys) & "<br<br>" &_
ucase(spiderLinksText) & "<br>" &_
spiderLinksHtml
end if
end function
%>
Using the code
Now we have a little beautiful new piece of code to use. But HOW? Well,
first of all we take our beautiful page to make "spider friendly"
and include the library. Second of all we place some content in the variables,
third of all we use the code.
Ok, a little hands-on code. Let's suppose this is your page:
<HTML>
<HEAD>
<TITLE>My Little Page</TITLE>
</HEAD>
<BODY>
Welcome to my page. This is my little page.<br>
A lot of stuff<br>
<IMG src="abeautifulimage.gif">
</BODY>
</HTML>
After the simple "restyle" it will be something like:
<!--#include file="cheating.asp"-->
<%
' The spiderWelcomeText will write a "Hello Spider"
' text in the dedicated paragraph. Sort of
' making a fool of the spider, or better clarifying
' what these things are to accidental non spider
' folks with very strange browsers.
spiderWelcomeText = "Hello, little dear spider!"
' The spiderMetaKeys is a comma-separated list of words
' the function will use to generate the text used for
' cheating the spider
spiderMetaKeys = "key1, key2, key3, key4"
' The spiderLinksText will write a friendly text
' before more important links to be added
spiderLinksText = "dear spider, look here, please..."
' The spiderLinksHtml will contain the HTML code for the Links
spiderLinksHtml = "<a href='1.asp'>Link1</A> - <a href='2.asp'>Link2</A>"
%>
<HTML>
<HEAD>
<TITLE>My Little Page</TITLE>
</HEAD>
<BODY>
<%=spiderContent()%>
Welcome to my page. This is my little page.<br>
A lot of stuff<br>
<IMG src="abeautifulimage.gif">
<%=spiderContent()%>
</BODY>
</HTML>
"Live" example
If you remember, in the isSpider function we have added a line to handle
the "tests". This line will let you see the spider's view of
http://www.yoursite.com/pageYouModified.asp simply by writing
http://www.yoursite.com/pageYouModified.asp?spider=1 .
Now let's take a look at Lastknight Dot Com's own implementation.
My declarations are a little more complex:
<%
' Spider cheating
spiderWelcomeText = "<table width=50% align=center><tr>" &_
"<td><B>Hello Spider! Welcome at www.lastknight.com</B>"
spiderMetaKeys = "lastknight, lk, lk.com, lastknight.com," &_
"lastknight.org, lastknight.net, lastknight, matteo, " &_
"matteo flora, matteo g.p. flora, matteo ""lk"", " &_
"programming, asp cheating, google chating, tutorials, " &_
"photos, milano, milan, neural networks, NN, asp.net, asp, " &_
"Active Server Pages, lastknight, LK, last knight, last, " &_
"lastknight dot com, lastknight dot org, lastknight dot net"
spiderLinksText = "<B>Dear spider, here is a list of" &_
"contents for you to handle...</B>"
spiderLinksHtml = "<a href='default.asp'>Home Page</a> - " &_
"<a href='contacts.asp'>Contacts</a> - " &_
"<a href='titorials.asp'>Tutorials</a> - " &_
"<a href='me.asp'>About me</a> - " &_
"<a href='blogger.asp'>Blogger</a> - " &_
"<a href='index.xml'>Blogger XML</a> - " &_
"<A href='/palm/'>Palm programming</A></td>" &_
"</tr></table>"
%>
What have I done? Well, basically only adding a TABLE for better formatting the
output. You can look at the result of all this work in any page of the site by adding
the ?spider=1 query-string.
Conclusions
Well, now you've got a powerful way to improve your website. Take it and let me
know of improvements and/or good implementation.
Ah, I almost forgot: all the souce code and the tutorial comes
to you subject to GNU Free Documentation License. Take a look
at it before using the code. Thanx.
