scripting:bash:hosts
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| scripting:bash:hosts [2015/04/09 21:48] – created warnaud | scripting:bash:hosts [2021/12/29 21:03] (current) – ↷ Links adapted because of a move operation warnaud | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | I always want to get rid of the sh*tload of ads/ | ||
| + | I have a solution on the DNS with Bind (using only one url to get an hosts file) see [[os: | ||
| + | I now use 2 DNS at home. One running Bind with default config. Another running dnsmasq for my home dns entry and the ease to add all those funky domain to hell.\\ | ||
| + | \\ | ||
| + | I use 2 scripts, one to generate a file containing all domains to blacklist. And a second one to merge it with my DNS entries + whitelist some of them.\\ | ||
| + | Structure is as follow: | ||
| + | * gen_domain_list.sh | ||
| + | * gen_hosts-dnsmasq.sh | ||
| + | * dl/ (folder containing the downloaded host files - erased before each run) | ||
| + | * list/ | ||
| + | * dns.list | ||
| + | * white.list | ||
| + | <code bash> | ||
| + | #! /bin/sh | ||
| + | # Script that generate a list of domain to blacklist | ||
| + | |||
| + | # First rm previous version of downloaded and generated files | ||
| + | echo ">> | ||
| + | rm dl/* | ||
| + | |||
| + | # Get the hosts files | ||
| + | echo ">> | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | wget -O dl/ | ||
| + | |||
| + | |||
| + | # Process files | ||
| + | echo ">> | ||
| + | for host_file in `ls ./dl/*.txt` | ||
| + | do | ||
| + | echo " | ||
| + | echo " | ||
| + | dos2unix $host_file | ||
| + | echo " | ||
| + | # Removes comments and empty lines | ||
| + | sed -i ' | ||
| + | sed -i '/ | ||
| + | # Replaces tabulation by space | ||
| + | sed -i "s/\t/ /ig" $host_file | ||
| + | # Removing lines containing ::1 localhost broadcasthost | ||
| + | sed -i '/:: | ||
| + | sed -i '/ | ||
| + | sed -i '/ | ||
| + | # Removing 0.0.0.0 | ||
| + | sed -i ' | ||
| + | # Removing 127.0.0.1 as well as mistakes like 127.0.1.1 | ||
| + | sed -i ' | ||
| + | # Removing trailing space | ||
| + | sed -i 's/ *$//' $host_file | ||
| + | # Send the result to a temp file | ||
| + | cat $host_file >> dl/ | ||
| + | done | ||
| + | |||
| + | echo ">> | ||
| + | # remove the www. in front of domains | ||
| + | #sed -i ' | ||
| + | # Lowercase everything | ||
| + | tr ' | ||
| + | # sort unique domains | ||
| + | sort -u -o hosts.sec dl/ | ||
| + | echo ">> | ||
| + | |||
| + | </ | ||
| + | |||
| + | <code bash> | ||
| + | #!/bin/sh | ||
| + | # Script that generate a hosts file for dnsmasq (first DNS entries then blacklisted domains) | ||
| + | # requires dns.list dns list | ||
| + | # requires white.list domain white listed | ||
| + | |||
| + | rm hosts hosts.add | ||
| + | |||
| + | # Concatenating files | ||
| + | echo " | ||
| + | cat list/ | ||
| + | echo " | ||
| + | comm -23 hosts.sec list/ | ||
| + | # Add 0.0.0.0 to all the lines | ||
| + | sed -i -e ' | ||
| + | # Add all those #%$@ domains to the hosts file | ||
| + | cat hosts.add >> hosts | ||
| + | echo " | ||
| + | |||
| + | </ | ||
| + | |||
| + | Content of dns.list | ||
| + | <code bash> | ||
| + | 127.0.0.1 | ||
| + | ::1 | ||
| + | fe00:: | ||
| + | ff00:: | ||
| + | ff02:: | ||
| + | ff02:: | ||
| + | |||
| + | |||
| + | #network devices and Services [1-30] | ||
| + | 192.168.1.1 | ||
| + | 192.168.1.2 | ||
| + | 192.168.1.3 | ||
| + | </ | ||
| + | |||
| + | Content of white.list | ||
| + | <code bash> | ||
| + | 24heures.ch | ||
| + | anibis.ch | ||
| + | beagleboard.org | ||
| + | billionuploads.com | ||
| + | bluewin.ch | ||
| + | fdj.fr | ||
| + | </ | ||
| + | And voilà :) | ||