Specialized Search engine optimization: Technical SEO refers to Site and server optimization that helps the crawler in crawling, indexing and ranking functions to rank your internet site improved. txt file is then parsed and may instruct the robotic as to which webpages aren't to generally be crawled. For a online https://www.youtube.com/watch?v=HmdZ6Ymv04I