tldextract: Timeout: The file lock \'some/path/to/8738.tldextract.json.lock\' could not be acquired

Question

I've been using tldextract for a while in my multiprocess, multithreaded script and it never caused any problems.

Now, it's giving me this error message and I've got no idea where to start looking for the root cause.

The line that actually crashes:

File "/home/user/anaconda3/lib/python3.8/site-packages/tldextract/tldextract.py", line 296, in extract
    return TLD_EXTRACTOR(url, include_psl_private_domains=include_psl_private_domains)

It is possible that the timeout is linked to the increased number of threads that I am using, however, I don't see why tldextract needs to lock anything anyway since the files that the error mentions are supposed to be read-only data files (I think).

In any case, is there a way to increase this timeout perhaps?

Answer 1

The file that the lock is supposed to protect is the TLD list. it locks it to update it. for me the goal is to avoid locks in any cost in a multithreaded system. but I am not fully familiar with what the developers tried to achieve with such an expensive mechanism (in term of running time).

You can try to solve the issue in a few ways:

1. Maybe The path, the process tried to access is not accessible to it for some reason (check your file system permission).
2. Don't use a live snapshot of the cache:

   # extract callable that falls back to the included TLD snapshot, no live HTTP fetching
   no_fetch_extract = tldextract.TLDExtract(fallback_to_snapshot=True)
   no_fetch_extract('http://www.google.com')
   

   3. set each thread to write to a different location

   # extract callable that reads/writes the updated TLD set to a different path
   custom_cache_extract = tldextract.TLDExtract(cache_file='/path/to/your/cache/file')
   custom_cache_extract('http://www.google.com')
   

   4. Don't use cache

   # extract callable that doesn't use caching
   no_cache_extract = tldextract.TLDExtract(cache_file=False)
   no_cache_extract('http://www.google.com')
   

Answer 2

Updating TLDExtract did the trick for me.