Showing posts with label Responsive website. Show all posts
Showing posts with label Responsive website. Show all posts
Today i am sharing this article especially helpful for site owner and developers as well 
 " Stop Search Engines Google, Yahoo, Amazon etc. from Crawling  Website ".
In order for your website to be found by other people, search engine crawlers, also sometimes referred to as bots or spiders, will crawl your website looking for updated text and links to update their search indexes.
Control search engine crawlers with a robots.txt file
Website owners, can instruct search engines on how they should crawl a website, by using a robots.txt file.
When a search engine crawls a website, it requests the robots.txt file first and then follows the rules within.
It's important to know robots.txt rules don't have to be followed by bots, and they are a guideline.
For instance to set a Crawl-delay for Google this must be done in the Google Webmaster tools.
For bad bots that abuse your site you should look at how to block bad users by User-agent in .htaccess.

Edit or create robots.txt file

The robots.txt file needs to be at the root of your site. If your domain was example.com it should be found:
On your Website:
http://example.com/robots.txt
On Your Server:
/home/userna5/public_html/robots.txt
You can also create a new file and call it robots.txt as just a plain-text file if you don't already have one.

Search engine User-agents

The most common rule you'd use in a robots.txt file is based on the User-agent of the search engine crawler.
Search engine crawlers use a User-agent to identify themselves when crawling, here are some common examples:
Top 3 US search engine User-agents:
Googlebot
Yahoo! Slurp
bingbot
Common search engine User-agents blocked:
AhrefsBot
Baiduspider
Ezooms
MJ12bot
YandexBot

Search engine crawler access via robots.txt file

There are quite a few options when it comes to controling how your site is crawled with the robots.txt file.
The User-agent: rule specifies which User-agent the rule applies to, and * is a wildcard matching any User-agent.
Disallow: sets the files or folders that are not allowed to be crawled.
Here are some of the most common uses of the robots.txt file:
 Set a crawl delay for all search engines:
If you had 1,000 pages on your website, a search engine could potentially index your entire site in a few minutes.
However this could cause high system resource usage with all of those pages loaded in a short time period.
A Crawl-delay: of 30 seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours
A Crawl-delay: of 500 seconds would allow crawlers to index your entire 1,000 page website in 5.8 days
You can set the Crawl-delay: for all search engines at once with:
User-agent: *
Crawl-delay: 30

How to Allow all search engines to crawl website?
By default search engines should be able to crawl your website, but you can also specify they are allowed with:
User-agent: *
Disallow:
How to Disallow all search engines from crawling website?
You can disallow any search engine from crawling your website, with these rules:
User-agent: *
Disallow: /

 Disallow one particular search engines from crawling your website?
You can disallow just one specific search engine from crawling your website, with these rules:
User-agent: Baiduspider
Disallow: /
 How to Disallow all search engines from particular folders?
If we had a few directories like /cgi-bin/, /private/, and /tmp/ we didn't want bots to crawl we could use this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /private/
Disallow: /tmp/

How to Disallow all search engines from particular files?
If we had a files like contactus.htm, index.htm, and store.htm we didn't want bots to crawl we could use this:
User-agent: *
Disallow: /contactus.htm
Disallow: /index.htm
Disallow: /store.htm
 How to Disallow all search engines?
If we only wanted to allow Googlebot access to our /private/ directory, and disallow all other bots we could use
:

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow:

HTML Markup
The HTML Markup consists of an ASP.Net GridView. I have applied the CSS class footable to the GridView.
<asp:GridView ID="GridView1" CssClass="footable" runat="server" AutoGenerateColumns="false"
    Style="max-width: 500px">
    <Columns>
        <asp:BoundField DataField="Name" HeaderText="Customer Name" />
        <asp:BoundField DataField="Id" HeaderText="Customer Id" />
        <asp:BoundField DataField="Country" HeaderText="Country" />
        <asp:BoundField DataField="Salary" HeaderText="Salary" />
    </Columns>
</asp:GridView>


Namespaces
You will need to import the following namespaces.

C#

using System.Data;

VB.Net
Imports System.Data


Binding the GridView
The GridView is populated using some dummy records using DataTable, you can populate it from database too.
After the GridView is populated with data you will need add the following attributes to each Header Column of GridView depending on its significance in different displays.
1. data-class = “expand” – You need to specify this attribute to the first column so that it shows the expand collapse button which will be used to show or hide the hidden column fields in Mobile Phone or Tablet.
2. data-hide = “phone” – You need to specify this attribute for all the columns that you want to hide in smaller displays like Mobile Phone or Tablet. The below table displays the possible values.
Value
Hidden in Mobile Phone
Hidden in Tablet
Hidden in Desktop
phone
Yes
No
No
phone,tablet
Yes
Yes
No
all
Yes
Yes
Yes

Also you need to set the GridView Header Row TableSection property to TableHeader so that it renders the HTML Table with THEAD and TBODY tags as it is necessary to have these tags to make the plugin work.
C#
protected void Page_Load(object sender, EventArgs e)
{
    if (!this.IsPostBack)
    {
        DataTable dt = new DataTable();
        dt.Columns.AddRange(new DataColumn[4] { new DataColumn("Id"), new DataColumn("Name"), new DataColumn("Country"), new DataColumn("Salary") });
        dt.Rows.Add(1, "John Hammond", "United States", 70000);
        dt.Rows.Add(2, "Mudassar Khan", "India", 40000);
        dt.Rows.Add(3, "Suzanne Mathews", "France", 30000);
        dt.Rows.Add(4, "Robert Schidner", "Russia", 50000);
        GridView1.DataSource = dt;
        GridView1.DataBind();

        //Attribute to show the Plus Minus Button.
        GridView1.HeaderRow.Cells[0].Attributes["data-class"] = "expand";

        //Attribute to hide column in Phone.
        GridView1.HeaderRow.Cells[2].Attributes["data-hide"] = "phone";
        GridView1.HeaderRow.Cells[3].Attributes["data-hide"] = "phone";

        //Adds THEAD and TBODY to GridView.
        GridView1.HeaderRow.TableSection = TableRowSection.TableHeader;
    }
}

VB.Net
Protected Sub Page_Load(sender As Object, e As EventArgs) Handles Me.Load
    If Not Me.IsPostBack Then
        Dim dt As New DataTable()
        dt.Columns.AddRange(New DataColumn(3) {New DataColumn("Id"), New DataColumn("Name"), New DataColumn("Country"), New DataColumn("Salary")})
        dt.Rows.Add(1, "John Hammond", "United States", 70000)
        dt.Rows.Add(2, "Mudassar Khan", "India", 40000)
        dt.Rows.Add(3, "Suzanne Mathews", "France", 30000)
        dt.Rows.Add(4, "Robert Schidner", "Russia", 50000)
        GridView1.DataSource = dt
        GridView1.DataBind()

        'Attribute to show the Plus Minus Button.
        GridView1.HeaderRow.Cells(0).Attributes("data-class") = "expand"

        'Attribute to hide column in Phone.
        GridView1.HeaderRow.Cells(2).Attributes("data-hide") = "phone"
        GridView1.HeaderRow.Cells(3).Attributes("data-hide") = "phone"

        'Adds THEAD and TBODY to GridView.
        GridView1.HeaderRow.TableSection = TableRowSection.TableHeader
    End If
End Sub


Applying the FooTable responsive Table jQuery Plugin
You just need to inherit, jQuery, Footable jQuery Plugin and Footable CSS files and inside the jQuery document ready event handler, apply the plugin to the GridView control.
<link href="https://cdnjs.cloudflare.com/ajax/libs/jquery-footable/0.1.0/css/footable.min.css"
    rel="stylesheet" type="text/css" />
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery-footable/0.1.0/js/footable.min.js">
</script>
<script type="text/javascript">
    $(function () {
        $('[id*=GridView1]').footable();
    });
</script>


Screenshots
GridView displayed in desktop


GridView displayed in when browser width set to Mobile Phone width