The content of the article
You read it right: I installed a data center at home and became my own provider. Not that I woke up one morning and said: “What should I do today? I will become my own Internet provider. ” Everything happened gradually. Like in the fairy tale “Porridge from the Ax” – at some point I looked and realized: damn it, but I made my own data center! And if you ask: “Why?”, I will answer simply: “Why not?”
This is a translation of an article, first published on the Hyperion Gray blog. Told Jason Hopper, recorded David Towler, translated Alyona Georgieva.
In fact, the main reason was that I scan the Internet a lot. This is not some dubious scan, but quite cool. In particular, I scan every IP deep on the Internet – and I do it extremely quickly. Which, of course, gives a very serious load on my network – and on iron, too. Scanning through a traditional provider is almost impossible: usually they have strict limitations, but, on the contrary, do not have sufficient bandwidth. There are, of course, services that allow you to use the database, but with them such a voluminous scan becomes too expensive – and quite quickly.
I had 250 thousand dollars in the account of one VSP provider, and even with a time limit (how I came to this is a separate big story). And when this money ended – and it ended, and quickly – I had to look for some other option.
I began to think up something else and after some calculations I found that it was cheaper to buy a server in my situation – so much so that in the end it paid off in about six months. Most companies think it’s better to use a cloud provider — or maybe they just don’t want to figure it out — and just pay the bills on time. But for technology companies like ours, who need not only to host a site, this is not an option.
"What is a process?"
Since no increased secrecy was required, I went the official way (wah-wah!). Turned to the Canadian Radio, Television and Telecommunications Commission (CRTC), the Canadian counterpart of the United States Federal Communications Commission (FCC), to obtain a basic license for international telecommunications services (BITS) – in general, there are a lot of acronyms.
Editor’s Note: in Russia, the functions of organizations designated by these abbreviations are performed Roskomnadzor. If you suddenly decide on an unprecedented feat, look first of all for two licenses – for telematic services and for data transfer, except for data transfer for voice information purposes (that is, without IP-telephony, you need a separate license for it).
It took some legal action, a notary and paperwork in triplicate, but did not have to go around on his head – in general, this is quite feasible. Oh, I almost forgot – there is still a period of public discussion of the license application, when people can write what they think about this. I have not received a single comment … (I burst into tears.)
When the legal training was successfully completed and I received official permission to sell the Internet, the fun began. I got the right – but I had nothing to sell. The process itself nobly frustrated me, and the cost of the Internet, which I could resell, was very high. And what to do?
Okay, a few years ago I met the owner of an Internet service provider, and he told me about the “wholesale” Internet from a large Canadian provider, Bell. At the same time, Bell itself does not apply to this service – it is understandable: why give away for nothing what can be sold at a higher price? True, there was one serious problem: in order to retail the “wholesale" Internet, it was necessary to stretch the fiber from a large access point to my house. And this is additional construction, permits from the city and a bunch of other overhead costs – in general, the monthly cost went beyond reasonable limits. And if you take into account that the contract must be concluded for at least six years …
Better shoot … Quoting Axl's immortal words. Rose, "where are we now?"
Another option: one Internet reseller from Bell had a significantly more affordable, but still expensive offer. I know what you’re thinking right now: “Hey Jason, why don’t you just rent a rack in a colocation center?” Good question. I tried, but each new server noticeably increases the rent. Everything was decent with the network, but I had to share it with other people – which means that if I or they have some kind of software bug, the whole network could collapse, and this will not please anyone.
Among other things, it is still very expensive – and I would overpay for functions that I am not going to use. Such a plan would be viable if I had a fixed set of hardware requirements and I did not change anything. But therein lies the snag: my scan along with many other operations will not be static; on the contrary, it will be more than dynamic. So I would have to constantly change the settings and functions – and this is such a headache that it is not worth it. But that's not all deep breath: The collocations say that they work 99.99999% of the time without fail – this indicator is called "five nines." But if you are ready to start the server yourself and agree to 99.9% – an amazing difference! – you can save some money. Well, you don’t have to overpay for unwanted functions included in the price.
“What have you done?”
Everything and myself.
He figured out how to build his own network, assembled a build, supplemented it with a network diagram and put it on a discussion in a subreddit / r / homelab. And yes, my nickname on Reddit is really 420SwagBootyWizzard – deal with it.
We move on: there are five main components in the network, all of them have names from “Startrek”.
USS Defiant – Dell R710, the primary messaging gateway. Places and analyzes all my data, and also manages them. Considering that I collect data all over the Internet, quite a lot of work hangs on this server.
Picard – My deep learning installation, which has several graphic processors (GPUs) and the most powerful CPU in the data center. Its main function is to be a deep learning and analysis machine. It saves a lot of time when I work on another project – analytical or machine learning.
Janeway – A very powerful solid state drive (SSD) database. It contains several "enterprise level" SSDs. The database must have a very high write speed in order to keep up with the incoming data, and have a multi-terabyte storage. Build this server was quite difficult. It should not only quickly record, but also quickly read, withstand intense recording, and also have a large capacity. Satisfying all these requirements was not easy. I calculated that a regular SSD (like Samsung Evo 860) is likely to die in a year of use.
Next comes Delta flyerAnother Dell R710. This is the server of my virtual machine, which runs the Proxmox environment.
And finally Databanks (I planned to name it in honor of the on-board computer of the Enterprise, but I could not find the name – if someone tells me, I will be grateful). It mainly works as a DNS server and a network drive (NAS) from Synology. It has a very large capacity and is mainly used for backups and archiving – but it can be hooked up to any virtual machine or other server that needs to store several large folders.
If you want to know more about DNS servers and their use in cyber attacks, check out My blog.
At the same time, we rebuilt our house – in the sense, I am like this: kamon, this is the best moment to add some more wiring! Before that, my wife thought that I was just a little more nuts than usual – but now she understood and appreciated everything. After repairing the house and laying new wiring, I went even further – and went broke into a glazed data center and a couple of business-class Wi-Fi access points. The result was a very small room – but a lot of space for my needs.
I am sure you already appreciated how significant the costs are: servers, chassis, guides, power distribution, uninterruptible power supply (just a big battery – so that the computers continue to work during power outages) and so on. Everything grows like a snowball – and pretty quickly. But, on the other hand, since I am now a technically licensed Internet service provider, I can legally sell the Internet to my neighbors – which, of course, is pretty vile. However, it is much more important that all this can facilitate my research in a rather efficient and economical way. And although the whole project is expensive, in the long run, everything is quite cost-effective and can really fight off in the first few years. And I know that I jumped up twenty levels of coolness right now – but we will try to control ourselves, right? 😛
“What are you going to do next?”
Since the work is not yet completed, I cannot build a clear picture of the future. I believe that I will develop and develop my project, because, to be honest, I like to be my own provider – you can even say, the king of my own domain. Now I have the freedom, speed and bandwidth to scan what I like and as often as I want. I am not saying that you should definitely follow my example, I only say that you could. It was surprisingly easy and a lot of fun.
One of the first ideas was to offer this way to the researchers of the information section, because, for example, we are already doing this, and now that the project is up and running, we definitely want to offer it to those who are professionally researching information security. When I found out how difficult it is to research the information security of large cloud providers, I thought: it would be a good idea to support research by providing access to shared datasets and collaborative research. And although since then we have moved away from this idea a bit due to the peculiarities of logistics and customer management, this definitely remains an option for the future.