Key Takeaways
From our discussions with RBL providers and our DAAR contractor, we gained valuable insights relevant to our community, especially those measuring DNS abuse.Interconnected Data Dependencies: Anyone experienced with large-scale data pipelines knows that such issues are common. Data pipelines often face a domino effect; a change in one component can impact the entire system. Fortunately, we did not have one RBL influencing others, and we detected the unusual pattern early, preventing further confusion.
Understanding RBL Use Case: RBL data providers collect data at scale. As noted in our OCTO document (OCTO-37), clients use this data for various purposes, such as whitelisting, spam filtering, block listing, and occasionally for abuse metrics. Each client focuses on their specific needs, often overlooking the broader ecosystem, unlike ICANN. Consequently, they lack the awareness and incentive to inform us about changes in their collection or retention policies. For us, even minor changes are significant as we study the DNS abuse landscape over time. Most clients neither notice nor care about these changes. While we clarify how our use cases differ from theirs, it's our responsibility to use their data properly.
Data Granularity Challenges: Given the reasons above and the vast amount of data generated daily, RBL providers lack the capacity to store long-term, granular data. Some update their feeds every 15 minutes, complicating fine-grained historical investigations. To address this, we needed to cross-reference three independent data sets: ICANN's internal RBL data, CyberToolBelt's, and the RBL providers.
read more
