Many of our clients offer whitepaper PDFs on their site to generate leads. Therefore, our clients don’t want people to find their whitepaper PDFs from a Google search. Rather, they want to get people’s email address first before giving access to their whitepaper. There are multiple ways to hide or No Index the specific file in WordPress which in my case, I use x-robots tag inside .htaccess file to hide the pdf:
<FilesMatch “Bizzflow-whitepaper-DEF.pdf”>
Header set X-Robots-Tag “noindex, noarchive, nosnippet”
</FilesMatch>
Why use x-robots tag instead of robots.txt:
The robots.txt does not prevent your page or file from being listed in search results. What it does is stop the bot from crawling your page, but if a third party links to your PDF file from their website, your page will still be listed.
If you stop the bot from crawling your page using robots.txt, it will not have the chance to see the X-Robots-Tag: no index response tag. Therefore, never disallow a page in robots.txt if you employ the X-Robots-Tag header.