{"id":196,"date":"2021-01-11T21:33:00","date_gmt":"2021-01-12T02:33:00","guid":{"rendered":"http:\/\/sycured.127.0.0.1.sslip.io\/?p=196"},"modified":"2024-01-13T19:57:48","modified_gmt":"2024-01-14T00:57:48","slug":"rbl-checker-refacto","status":"publish","type":"post","link":"http:\/\/10.42.0.68:8080\/blog\/rbl-checker-refacto","title":{"rendered":"RBL-Checker: details on the refactorization"},"content":{"rendered":"\n<p>A long time ago, I wrote <a href=\"https:\/\/github.com\/sycured\/rbl-checker\">RBL-Checker<\/a>, a tool to check if a range IP is blacklisted or not.<\/p>\n\n\n\n<p>I continue to use it but need a lot more complete and that integrate better in my use case.<\/p>\n\n\n\n<p>In this post, you&#8217;ll learn:<\/p>\n\n\n\n<ul>\n<li>my use case (actual and future)<\/li>\n\n\n\n<li>my choice of style for writing code and tools<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">My use case<\/h2>\n\n\n\n<p>I wrote the first version at home when I worked for a hosting provider. This company never sees the ROI to build this style of little software.<\/p>\n\n\n\n<p>I can&#8217;t tell you that I liked this manager when I needed to debug one email server with a very high queue with nothing about metrology, detection of a blacklist, exclusively reading the log without an ELK or similar.<\/p>\n\n\n\n<p>Right now, I continue to have my email servers so I use my software but also for any clients that need to monitor and alert on them.<\/p>\n\n\n\n<p>My future use case can be for a non-profit IAP because it&#8217;s very important to keep range IP clean and be alerted when an IP becomes blacklisted.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">My choice<\/h2>\n\n\n\n<p>My first version was in <a href=\"https:\/\/www.python.org\/\">Python<\/a> and I continue in that way.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">From monolith to 2 services<\/h3>\n\n\n\n<p>I moved from a service that uses subprocess to do check-in concurrency to 2 services because I need to check very quickly a large range of IP.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Consumer<\/h4>\n\n\n\n<p>Asynchronous service taking a message from the queue and checking IP across <a href=\"https:\/\/github.com\/sycured\/rbl-checker\/blob\/main\/consumer\/rbls.py\">the list<\/a>.<br>Finally, add blacklisted IP with the blacklist name and DateTime of the check to the database.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Rest API<\/h4>\n\n\n\n<p>I use it exclusively to add range to publish the message to the queue.<br>Maybe later, I&#8217;ll add an endpoint to know how many messages are in the queue.<\/p>\n\n\n\n<p>The biggest move can be a migration to Haskell but right now, I don&#8217;t need it.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Possible big change<\/h4>\n\n\n\n<p>Migrating from JSON to binary format and in that case, I see these possibilities:<\/p>\n\n\n\n<ul>\n<li><a href=\"https:\/\/avro.apache.org\/docs\/current\/\">Avro<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/google.github.io\/flatbuffers\/\">Flatbuffers<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/msgpack.org\/index.html\">MessagePack<\/a><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">The stack<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">AIOPG<\/h4>\n\n\n\n<p><a href=\"https:\/\/aiopg.readthedocs.io\/en\/stable\/\">AIOPG<\/a> is asynchronous compared to <a href=\"https:\/\/www.psycopg.org\/\">psycopg2<\/a>.<\/p>\n\n\n\n<p>I use <a href=\"https:\/\/www.postgresql.org\/\">PostgreSQL<\/a> and <a href=\"https:\/\/www.yugabyte.com\/yugabytedb\/\">Yugabyte<\/a>, both can be used with PostgreSQL&#8217;s drivers so it was my choice to stay with <em>classic<\/em> database that can be pushed in production very easily and won&#8217;t be a pain.<\/p>\n\n\n\n<p>Also, public cloud providers provide managed PostgreSQL (like <a href=\"https:\/\/aws.amazon.com\/rds\/postgresql\/\">Amazon RDS for PostgreSQL <\/a>) so it&#8217;s easy to deploy.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">FastAPI<\/h4>\n\n\n\n<p>One year ago, I left <a href=\"https:\/\/palletsprojects.com\/p\/flask\/\">Flask<\/a> for <a href=\"https:\/\/fastapi.tiangolo.com\/\">FastAPI<\/a>.<\/p>\n\n\n\n<p>It permits to build async rest api and provides <a href=\"https:\/\/swagger.io\/specification\/\">OpenAPI<\/a> with <a href=\"https:\/\/github.com\/swagger-api\/swagger-ui\">Swagger UI<\/a> and <a href=\"https:\/\/github.com\/Redocly\/redoc\">ReDoc<\/a> by default.<\/p>\n\n\n\n<p>Deploying Python-based software it&#8217;s very easy and compatible with every CPU architecture (ARM, RISC-V, x86).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Kafka<\/h4>\n\n\n\n<p>I have one <a href=\"https:\/\/kafka.apache.org\/\">Kafka<\/a> cluster running so I continue to use my actual stack and I won&#8217;t add a new service (<a href=\"https:\/\/nats.io\/\">NATS<\/a> &amp; co) for this little job and nothing more.<\/p>\n\n\n\n<p>I activated <a href=\"https:\/\/cwiki.apache.org\/confluence\/display\/KAFKA\/Compression\">compression on the producer<\/a> (<a href=\"https:\/\/github.com\/lz4\/lz4\">lz4<\/a>) to gain disk space.<\/p>\n\n\n\n<p>Some cloud providers provide Kafka or Kafka compatible managed service so it&#8217;s very easy to deploy and you don&#8217;t need to manage it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A long time ago, I wrote RBL-Checker, a tool to check if a range IP is blacklisted or not. I continue to use it but need a lot more complete and that integrate better in my use case. In this post, you&#8217;ll learn: My use case I wrote the first version at home when I [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"saved_in_kubio":false,"footnotes":""},"categories":[28],"tags":[11,29],"_links":{"self":[{"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/posts\/196"}],"collection":[{"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/comments?post=196"}],"version-history":[{"count":1,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/posts\/196\/revisions"}],"predecessor-version":[{"id":197,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/posts\/196\/revisions\/197"}],"wp:attachment":[{"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/media?parent=196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/categories?post=196"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/tags?post=196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}