[OpenWrt-Devel] Slow DNSMasq with > 100, 000 entries in additional addresses file

[Dave Taht]

On Thu, Dec 29, 2016 at 8:09 AM, TheWerthFam <thewerthfam at gmail.com> wrote:
> Right now I'd rather not customize the code.  There are two directions I'm
> going to try first.
> Give unbound a try to serve DNS, keeping Dnsmasq for DHCP.  If that doesn't
> work try converting the list to a hosts file pointing to a local pixelsrv
> address.  There are some other blog posts that indicate that the hosts file
> can handle a lot more entries.  Like https://github.com/pi-hole/pi-hole
> Maybe just run pi-hole on openwrt.

Well, I've had a bit of fun feeding large blocklists into cmph. Using
the "chd" algorithm, it creates an index file from a 24MB blocklist
into a 800K one. (but you still need the original data and a secondary
index) I also fiddled a bit with bloom filters, which strike me as
appropo. It seems feasible to establish a large dataset of read-only
data with a fast index (that can be discarded in low memory
situations, rather than swapped out)

I'll take a look at pi-hole...

> Cheers
>    Derek
>
>
> On 12/28/2016 02:21 PM, Dave Taht wrote:
>>
>> On Tue, Dec 27, 2016 at 11:03 PM, TheWerthFam <thewerthfam at gmail.com>
>> wrote:
>>>
>>> Thanks for the feedback, I'll look into NFQUEUE.  I'm forcing the use of
>>> my
>>> dns by iptables.  I'm also using a transparent squid and e2guardian to
>>> filter content.  I like the idea of the dns based blacklist to add some
>>> filtering capabilities since I don't want to try and filter https types
>>> sites.  I know no solution in perfect.
>>
>> I've been thinking about this, and given the large amount of active
>> data in a very small memory space have been thinking that another
>> approach would be more fruitful. Convert the giant table into a
>> "minimally perfect hash", and mmap it into memory read-only, so it can
>> be discarded under memory pressure, unlike ipset, squid, or dnsmasq
>> based approaches.
>>
>>
>>> Cheers
>>>   Derek
>>>
>>>
>>>
>>> On 12/27/2016 01:53 PM, philipp_subx at redfish-solutions.com wrote:
>>>>>
>>>>> On Dec 26, 2016, at 10:32 AM, TheWerthFam <thewerthfam at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Using the adblock set of scripts to block malware and porn sites. The
>>>>> porn sites list is 800,000 entries, about 10x the number of sites
>>>>> adblock
>>>>> normally uses.  With the full list of malware and porn domains loaded,
>>>>> dnsmasq takes 115M of memory and normally sits around 50% CPU usage
>>>>> with
>>>>> moderate browsing usage.  CPU and RAM usage isn't really a problem
>>>>> other
>>>>> than lookups are slow now. Platform is cc 15.05.1 r49389 on banana pi
>>>>> r1.
>>>>>
>>>>> The adblock script takes the different lists, creates files in
>>>>> /tmp/dnsmasq.d/ entries looking like
>>>>> local=/domainnottogoto.com/   one entry per line.  The goal is to
>>>>> return
>>>>> NXDOMAIN to entries in the lists. Lists are sorted and with unique
>>>>> entries.
>>>>>
>>>>> I've tried increasing the cachesize to 10,000 but that made no change.
>>>>> Tried neg-ttl=3600 with default negative caching enabled with no
>>>>> change.
>>>>>
>>>>> Are there dnsmasq setting that will improve the performance?  or should
>>>>> it be configured differently to achieve this goal?
>>>>> Perhaps unbound would be better suited?
>>>>>
>>>>> Cheers
>>>>>      Derek
>>>>
>>>>
>>>> Not to rain on your parade, but the obvious defeat of this solution
>>>> would
>>>> be to point to an external website which does DNS lookups for you, and
>>>> then
>>>> edit the URL to have an IP address in place of the host name.
>>>>
>>>> I would use netfilter’s NFQUEUE and make a user-space decision based on
>>>> packet-destination (since it seems you’re filtering outbound traffic
>>>> requests).
>>>>
>>>> After all, it’s not the NAME you don’t want to talk to… it’s the HOST
>>>> that
>>>> bears that NAME.
>>>>
>>>> -Philip
>>>>
>>> _______________________________________________
>>> openwrt-devel mailing list
>>> openwrt-devel at lists.openwrt.org
>>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>
>>
>>
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel