- It’s hard to do the limiting N+1. Just go for caching.
Chat with Mark on Scraping
Watir: Web Application Testing in Ruby. Slow Internet – results in
Net::ReadTimeouterrors. Headless is faster, but you have to assign the user-agent. No prod for Watir since it leaks memory.
- Mechanize does not run JS. But it is faster than Watir.
- Check out the Resources log in Inspect Element to see the endpoints that make sense.