Robots.txt files tell Search Engines what should and should not be crawled
NOTE – This is very different from the Robots Meta Tag. The crawler will see this file before it tries to call the page, so this file will override the Robots Meta tags on the pages.
Robots.txt files should be stored in the root directory
Remember, the point of the robots.txt file is to exclude pages from being crawled. So if a page or directory is banned, it will never even get to see what code is on those page(s). Accordingly, no code of those pages could change the bots behavior to re-index the page. So, robots.txt will overwrite the meta and robots tags on the page.