It began with annoying hints in Visual Studio Code, but sometimes, those hints were what I was going to type. For the cost of a tab, I save 10+ keystrokes. Another cost may be in the copyright loss often associated with AI.

Then, AI output began to appear in search results when I was looking up math concepts to refresh my knowledge for assisting my daughter in their homework.

Finally, I used AI to mockup a specific looking photo image of a screen with a spreadsheet for an idea I had for a development competition. I submitted my application idea with the AI image initially, however, while it looked pretty, the text was all wrong in the spreadsheet. I used Sheets to generate the exact contents I needed for the image, created a PDF of the printed sheet and then commissioned a local artist to transform the flat spreadsheet to look like the reference image.

Claude

I used Claude to prototype a project (Autonomi Transaction Reporter). I reached roadblocks and had to start over numerous times, however I think I was getting better at getting the output that I wanted. I was able to make a rudementary solution in time for the competition deadline, however, I had already been disqualified due to an oveerexcited project supporter.

Base44

My daughter requested that I attend an event with her while dressed as the character Kinger from The Amazing Digital Circus. I needed to create a template that joins two cone segments to form the foundation of the head. I try with multiple online tools to generate the multi-page/blueprint size frustums I need, ultimately I don’t succeed. I’m capable of drafting manually, but I am feeling this is an opportunity to try generating a bespoke app.

Base44 came up in a search result for AI app generator. My 127 word prompt generated an app in about 5 minutes. The print output of the first result was scaled wrong, so I generated an update, which produced something that looks correct but still had some render defects. However, it was close enough for my purpose. I did make changes to the settings for three different paper versions before cutting the final pattern in plastic mesh.

After hot-glueing the panels together, and then forming the joined cones, I then built-up the rest of the crown-hat solution, including the cross, in mesh. My initial intent of hot-glueing paper to the mesh was an instant NO! due to nearly burning my fingers on the first attempt. I have low durability requirements for this one-time use costume (maybe I sell it after?). I’m nearly $16 into my $40 budget, which must include a cape and optional gloves. So I find wide masking tape that fits the color of the character close enough and cover the mesh in vertical strips.

The following glossary attempts to make this clearer:

• Autonomi - Provides a decentralized storage network that allows you to pay once to permanently store and then retrieve information free of charge.
• Faucet Dapp - Is a term used for a system that rewards users with free cryptocurrencies.

In our use case, “Testing Autonomi” means to upload data to the network, which requires two separate cryptocurrencies:

• ANT - Autonomi Network Token is the currency used to pay and reward nodes for uploading data.
• ETH - Ethereum is another currency we must pay to use ANT (called gas).

The code produced by this adventure can be found at this repo

Autonomi PrePost-launch

When this project started, the Real token had not launched (Feb 61114, 2025), so the only way to gain ANT had been to run nodes on the test networks, earning little bits along the way. Since the nodes are allocated randomly, it can be challenging to earn enough tokens to practically upload a file. For example, it takes roughly 12 rewards to earn enough ANT for the smallest file to upload, which can take more than a week to earn at the current rate.

To allow users to more quickly and easily test uploads, we needed a simple way to grant test tokens that have already been earned for others to use. After a discord member suggested that there be a faucet, I thought, “I can learn something even if I don’t succeed. I’ll give it a try!”

The setup

First up, I needed to find out how to transfer tokens in Python. ANT is something called ERC-20 that lives on a second-layer network called Arbitrum (which is compatible with Ethereum, a popular cryptocurrency.) I was confident that transferring ETH would be well documented, so I looked for Python examples that transferred ERC-20 tokens. After more searching and testing of code than I hoped, I finally stumbled on this page that just worked. A small bit of research into the Web3 API that the example was from showed the slight change needed to also send ETH from the command line.

That gets us most of the way there, the rest is wrapping it all into something accessable to others.

Getting started

I’ve used Python for simple task automation recently over using other previous tools like bash/perl/php as it provides a lot of power and is often installed on systems by default. I’ve also done enough with Python to know to use virtual environments to localize package installation. So I start with creating and activating a Python 3 environment to paste my example.

Topping up

I know about faucets, in part, because I have been using one to fund my own testing on the web3 stuff. Since their faucet has been reliable, I decided to use Alchemy’s API endpoints to power my transactions. This faucet will definitely fit within Alchemy’s free tier. To have some funds to play with, I topped up my wallet with Alchemy’s faucet. I had already been given some test ANT and ETH by the person who suggested the faucet, but I wanted to save that for after the faucet was working.

Linking to Alchemy

I logged into Alchemy and in my dashboard, I created an App endpoint connected to the Arbitrum Sepolia network that I used to fill in the endpoint from the example. To get the sample code to work, I also had to import the private key of my wallet. Since this code was running on my laptop, I wasn’t worried about the key leaking.

Finally, I used the ERC_20 address of the ANT token for the contract information in the sample code.

Flask

Now, outside of tutorials for other things, I’ve not done much front-end anything, mostly I have worked on the back-end of things. I knew I wanted to turn to Flask as a simple front-end to handle URLs, and an early hit found me an example to use to get started. I created my template html files and wired up the simple main.py to launch my form. All I needed was a text box for the wallet address and a submit button.

Once I could see my form data rendered, I could determine how to interact with the value as provided by Flask.

Building out the main function

Now to get to the meat of it. I created a function called drip_coins() that returns a payload with two items, “status” and “reason”. First I start with a fixed status of False and a ‘testing’ result. My goal is to pass the results to the render form. If the status is not “True” (everything else), render the ‘fail.html’ template which shows all the values returned. Once I have that wired up I begin to work the code from the transaction sample into drip_coins().

First up, there are asserts littered in the code. Anything that occurs after the app is launched would cause the app to crash, so move some of the sample code (like the part that loads environment variables) out of the function and into the main body of the code. That still leaves us with things that will be defined after the code launches, like the user’s wallet address.

We’ll add other validations before we get here, but one of the first things the code does is check the wallet address for a checksum (the way the upper and lowercase characters are defined in an address help to validate that the address is correct). If we fail the check here, return a failed response. I like to test edge cases along the way, so I try various good and bad wallet addresses to verify my check is good.

Next, we include the code up to the point that checks the token and ETH balances. This seems like a very reasonable checkpoint, let’s return a result showing the amount of token and ETH using the False status to display the values.

The sample code reported the chain ID of the blockchain, we can use that as another validation that things are correctly configured. Add a header defining the chain ID and return a False status if the values don’t match.

Instead of the user entering the amount to transfer, we can wire that to a variable we set in the header. In case we have to iterate a lot, use a small value (I used 0.00000001 to start and incremented the value between runs so I could tell the transactions apart on the blockchain reporter).

Now, we’re finally doing transactions. Return the ETH and ANT transaction IDs as False results.

That looks good. Let’s create a “success.html” to give a render target for our good result! Then, of course, we need to change the status of the good results to True so we can render our success page. :)

Validating data

It’s getting serious now, we have a function to protect. I created a validate_request() function and passed it the form_data. We know a wallet address can be between 0 and 40 characters long and only contain hexadecimal characters (upper and lowercase) and the letter x/X (to notate it is a hex number), so that’s our first check.

Persistence

A common feature of a faucet is either a one time use (per address) or a cooling-off period/activity required before the next drip. That requires some form of data persistence. Luckily, for our simple needs, sqlite3 is a robust tool that is included with Python.

After we define the database in the header, we need a way to know that the database is ready to go when we launch the server. So I created a prepare_faucet_db() function that tries to get the first row from the database. If that fails, we immediately try to set up the database for the first time by creating the structure and inserting a test record. I then invoke this function when the app first launches so we crash right away if we can’t get/create the database.

The next step is to check if the wallet already exists in the database (we know it doesn’t), so I created the check_db() function to select any records in the database that match the wallet ID. I start with automatically returning True so that I can test the failure that the wallet already exists and set an appropriate message in my reason. Then I add the real check, which returns a False if there is no match.

And finally, I created the add_db() function to add transactions to the database. This causes subsequent transactions for the same wallet to be declined.

Bringing it online

hcaptcha

Now it’s all fine and good to develop on a local computer, eventually we want others to test our creation, but I am running a little server that I don’t want a thundering herd by someone writing a script to drain the faucet’s wallet. To protect the ‘expensive’ database calls and other complex processing, I want to protect the input form. Some search for a privacy-compliant option found me hcaptcha, which required very little to set up. I had to register for the free service, create an endpoint, and add some JavaScript and a <div> tag to my web form template file. Hcaptcha also requires to be served from an HTTPS web page, so I had to set up that (described briefly below).

Following my normal steps, I created a check_hcapthca() function, set up a return False and a reason of the captcha failed, added the new function to validate_request. This showed me the new variable that was returned by the form. I added a check to make sure that the captcha was provided or we fail right away.

If the captcha is provided, we need to verify by making an API request to hcaptcha and getting confirmation that it’s both valid and timely. This payload is quite large, so I don’t have any filtering on it and it’s a potential vector for buffer overflows and wasting resources. Something for the future if this lasts more than a week/proof-of-concept. Again, I start with the failure case hard coded, then add the API call, and if we get a success finally add the True status.

HTTPS/SSL Let’s Encrypt

I prefer NGINX as a web gateway which has support for Let’s Encrypt using CertBot.

I log into a server with a public IP address (and a private address on my internal network), create a nginx configuration for http://ant.xd7.org, test my nginx configuration and reload nginx. Then I add the public address to DNS and wait a few minutes for the update to be published before starting Certbot to create an HTTPS certificate for my domain.

Now, I can change the upgraded configuration file to point to my Flask backend, check and reload nginx and I can access the faucet at https://ant.xd7.org!

Faucet Wallet

To get things going for real, I created a NEW wallet and transferred some initial funds from what had already been provided and then linked the wallet to the server by copying the private key over and ran a test transaction to my own wallet.

Testing

I announced in the Autonomi discord channel that the faucet was up for testing and a few people went ahead and submitted their wallet addresses successfully. No one reported any issues. Then the user who asked for the faucet tested and gave the go-ahead to increase the faucet to .25ANT and .001ETH, so I reset the database, and deployed the changes.

Gunicorn

I know that the built-in web server for Flask is not production ready (besides, it warns you every time you start the server), so I logged into the server, activated my Python virtual environment, installed gunicorn, and figured out how to launch my Flask app with gunicorn (which entailed refactoring a little to not pass the faucetdb as a parameter to the database calls as gunicorn was not creating the database object when importing the app).

We’re live!

But are we finished? Not one to leave a good thing alone, I wanted to check the faucets from my phone. But I don’t want to impact the live instance with my testing, so I create a development server on a different port and set up another NGINX reverse proxy to activate the new internal node.

Success/Drips

The success endpoint was pretty simple, it just counts the number of records (minus the initial test record) and displays it onscreen. The only difficult part was I had already used the ‘success.html’ template filename, so I renamed the endpoint from /success to /drips. The next day, I used CSS to increase the font size to very large, so it renders visibly on a smartphone.

Rate limit

After a little discussion, the lead user suggested a good idea for the endpoint… a Rate limit. I decided to start with 6 drips/hour as a reasonable velocity.

First, check_db() became too generic so was renamed to check_db_for_wallet(), and then a new function check_db_for_rate() was defined and set to automatically return a True. Then I added a simple query that checks the COUNT() of the number of transactions in the last RATE_WINDOW seconds is more than RATE_LIMIT in which case it returns the count as a True and False only if we are below the rate limit.

Restore test transactions

It bugged me that I reset the database and lost the initial ‘live’ transactions. I initially chose that route so the testers could re-request a full faucet drip, but deleting the database was not the only option. The solution was something I already planned on, but didn’t execute initially to stay focused on getting things testable.

So, I pulled up a copy of the database I had cloned early in my testing and created a Python list of transactions to re-insert. Now I have a record of all the transactions that the faucet has succeeded with. But I didn’t want the first users to be left out, which required one small change. In the query for existing transactions, I added TIME_HORIZON which is a unix timestamp of the last transaction from the test db, so only returns transactions newer than that horizon.

This could later be extended to allow the same wallet to request more than once with its own velocity (like once/week).

Limit upload content length

Flask provides for a simple way to block unwanted data, MAX_CONTENT_LENGTH, which will return a 413 HTTP Response when exceeded.

Parameterize SQL

A few of the queries were spiked with non-preferred syntax. I cleaned those up by parameterizing the statements.

Integrate MetaMask wallet

A feature I see in some dapps, is the ability to connect to a MetaMask wallet. Digging around I discovered this 3 year old example, which showed a very simple solution.

After getting the wallet address to auto-populate the form, I extended the code to turn the input field in to a drop down list if more than wallet address is connected.

Favicon

I am a little annoyed by the failed requests for the favicon.ico file, so I created a ‘static’ directory, uploaded a copy of my logo and set the image in the form header.

Now finished?

There isn’t a need to view transactions since they are all visible on the blockchain. Extending the faucet to have allow multiple drips doesn’t fit in the current use, so MAYBE we’re done… for now. :)

Donate Button!

I added the wallet address to the page, with a button to copy the address to your clipboard.

Adding transaction information

I accidentally committed some testing harness when bringing code from the dev environment, which created non-existent transactions while validating the live faucet. To make that transparent, I’m now adding the hash for both transactions.

If we detect that the test harness is active, record 0.0 for drip amounts since we didn’t do an actual transaction.

In this step, I created a migration sql file and also updated the broken transactions to 0.0 eth/ant.

Switching to the Mainnet!

Ok, on February 14th, 2025, the real ANT token launched on the Arbitrum One network. In theory this is simple.

• Change the network endpoint
• Change the network chain id
• Change the token contract to the live token
• Create a new wallet
• Fund the wallet
• Connect the new wallet
• Set new Drip rates
• Change the wallet donate address on the input form

All of which went swimmingly, however… transactions on the mainnet are confirming faster than the faucet issues them introducing a race condition on the nonce (serial number) of the second transaction.

I solved the broken response by wrapping the transactions in try/except blocks and I also wait for the first transaction to fully complete before issuing the second one.

Finally, I moved the ANT Token transaction first as we don’t want to waste ETH if we had a problem with the ANT.

Announce in the forum

We announced the faucet as live in the Autonomi forum, inviting people to either donate to or test the faucet. Both occurred, which was how we discovered the nonce bug above.

Increase wallet field size

Per a request in the forum, to help with people sending screenshots of the faucet, the width of the wallet field was increased to show the full wallet address.

I also reintroduced the Metamask integration, with corrections to allow Metamask to return lowercase wallet addresses and still work correctly.

Temporarily Halt Faucet

Well, it didn’t take long for the faucet to be abused (likely). However, I don’t have time to work on solutions at the moment, so to save funds I’m pausing the faucet. Ugh.

Add forum integration

I received permission from the Autonomi Community Forum to use the system as more user friction for people that want a drip.

To receive a drip, they need to enter a forum name that is over 2 days old and generate a MD5 hash of their membername and a secret phrase. Then they paste that hash as a reply to a specific forum topic, and copy the direct link to the post. Finally, they come back to the form and add the forum post link to the form with their membername and wallet and submit to receive a drip.

This may be enough friction to lift/increase the rate limit.

Badges soft launch

While integrating the forum validation, I announced that badges will be coming for wallets. The first badge, 1st just means you got a drip. Other badges I’m thinking of right now are:

• New wallet
• Old Wallet
• Cold Wallet (received drip but no activity in first 30 days)
• 1st upload
• 2nd upload
• 3rd upload (etc over first 30 days)
• Theft (funds swept to another wallet)
• Spare Change (returning funds too small to make an upload back to the faucet)
• Grower (adds funds to own wallet after receiving a drip)
• Patron (donates more in proceeds to faucet than dripped)

May 2026 - This historic blog refers to the Autonomi 1 network which is now offline, replaced with the Autonomi 2 network and different tooling.

Getting started

You will need the following:

• Current Autonomi link of Launchpad for Linux (Intel or ARM)
• An Arbitrum One Wallet address. MetaMask example here
• 35GB storage/node

User choice

There are two options for running your Launchpad:

sudo/root user

In this scenario, Launchpad will create a new user called ‘ant’, that will be used to run the worker nodes. This is safer than running the nodes as the super user.

non-root user

Here, Launchpad will spin up worker nodes as your specified user.

Storage location

By default, Launchpad creates the node information in ~/.local/share/autonomi

:warning: This has a side-effect that running nodes as sudo/root creates the nodes in /root/.local/share which the ant user does NOT have access to, and will crash when trying to run.

The two launchpad solutions are:

• run launchpad as a non-root user
• mount an additional drive/partition that the ‘ant’ user can access

Let’s Begin

Getting the client

wget https://node-launchpad.s3.eu-west-2.amazonaws.com/node-launchpad-0.5.8-x86_64-unknown-linux-musl.zip

Login to your system, and pull down the client using wget. If unzip is not installed by default, it is quick to grab…

sudo apt update
sudo apt -y install zip
sudo apt clean

We update the system packages information, install the zip package and clean the packages cache to save space.

unzip node-launchpad-0.5.4-x86_64-unknown-linux-musl.zip

Now we can unzip the launchpad and start it with ./node-launchpad.

Launchpad

This is our unconfigured Launchpad dashboard. As we make changes, we will automatically navigate back to the [S]tatus page. Use the [O] letter key to navigate to the Options screen.

This is the unconfigured Launchpad Options screen. From here we’ll make changes to setup our nodes.

Storage Drive

As mentioned previously, your instance will by default store data in the ~/.local/share folder, which is usually located on the primary disk/partition. This may not be ideal as it’s smart to have extra space for logs on the primary partition and to have our data mounted as a separate disk/partition.

To change the storage option, press [Control+D] and select the Drive.

As with most settings, changing the storage location will require any existing nodes to be reset. Press the [Enter] key to continue.

Network Connection Mode

Press the [Control+K] key.

Beyond Automatic, there are three Connection Modes to access the network.

• UPnP attempts to use Plug-n-Play to configure your router to connect to your nodes.
• Home Network uses relay nodes to send/receive traffic.
• Custom Ports allows you to define which ports your nodes use instead of being chosen automatically.

I’m going to use Custom Ports in this example, which means that I have to map the firewall/router ports to my instance. (A topic not covered here as it’s too broad to cover all the various routers available).

Here I’ve chosen port 50000 which selects the next 50 ports (the maximum that Launchpad allows).

Then we confirm the changes. And here Launchpad alerts us to setup port forwarding from our router.

Add your wallet

Our rewards for storing content are paid on the blockchain, this requires us to have a valid address if we want to get paid.

Press the [Control+B] key. First up, we need to agree to the Terms of Service by pressing [Y].

Then we enter a valid wallet address and continue with [Enter].

All Ready To Go

Now we can review our settings before we begin to launch nodes.

Press the [S] key to get back to the Status screen.

Press [Control+G] to begin manage nodes. We’ll start with 1 node, you can use the arrow keys to change the total number of nodes to manage. Press the [Enter] key to confirm.

And finally we can see the node running in Launchpad.

Depending on your own infrastructure needs, you may need to provision the cluster before you get the IP Addresses necessary for /etc/hosts.

This series uses a public network and an internal private network, as my internal network connections are 10GB and unmetered, which gives our inter-node communications a boost. For /etc/hosts, I used the internal ip addresses. In practice, my nodes still need an external ip address or default route to request packages/DNS/etc, but I have firewalled all inbound connections on the public networks and access the internal ip addresses using a bastion host that sits on both networks and lets ports 22/80/443 in. When I do this in production, my ssh interface is one of two nodes dedicated to cluster access and the 80/443 reverse proxies are seperate nodes without public ssh access.

In this post, I will take one other shortcut. I’m using the master nodes ip address for the control plane endpoint. In production, I would setup another, internal load balancer pointing to my multiple control plane nodes. This tutorial DOES setup the cluster to be upgradeable to high availability by adding more nodes (something to be covered in a future post).

Infrastructure

Initializing the cluster

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --control-plane-endpoint "xd7control:6443" --upload-certs --apiserver-advertise-address 10.7.222.12

Let’s break this down.

• We’re going to be using Calico which defaults to a pod network of 192.168.0.0/16, so we create the cluster that way.
• Control-plane-endpoint is something we can’t add to the cluster after it’s created, so we define it now using our /etc/hosts entry to make things work.
• Upload-certs is helpful if we’re going to be adding more control plane nodes in the next few hours (We can always restart the join process to add nodes after this window has expired).
• Apiserver-advertise-address is set to the internal address of the node since we want to use the internal network and kubernetes will, by default, pick an address on the network interface with the default route instead of our internal ip’s.

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Next, we’ll follow the instructions to clone the kubeconfig information to our hosts nonroot user. Through various means for ssh port forwarding that will be addressed in another post, you can also clone the kubeconfig to your local machine to have remote access to kubectl.

ubuntu@imaster00:~$ kubectl get nodes -o wide
NAME        STATUS     ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
imaster00   NotReady   control-plane   5m    v1.30.5   10.0.0.12   <none>        Ubuntu 24.04 LTS   6.8.0-31-generic   containerd://1.7.22

And we test kubectl

Calico

Kubernetes needs a Container Network Interface as a layer of abstraction. While it’s common to see Calico as the CNI, most tutorials specify just the raw manifests to get a cluster up and running instead of using the full control operator, this is mentioned in the Calico website:

Calico can also be installed using raw manifests as an alternative to the operator. The manifests contain the necessary resources for installing Calico on each node in your Kubernetes cluster. Using manifests is not recommended as they cannot automatically manage the lifecycle of the Calico as the operator does. However, manifests may be useful for clusters that require highly specific modifications to the underlying Kubernetes resources.

Since we don’t have any specific needs other than wanting to use the internal network, and we want a cluster we can upgrade, we’ll install the operator and then request the network to be provisioned.

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/custom-resources.yaml 

kubectl get pods -A -w

We can monitor the network coming up as we see first the calico-system and then the calico-apiserver pods go through the phases until they reach Running. Hit Control+C to exit the watch loop.

ubuntu@imaster00:~$ kubectl get nodes
NAME        STATUS   ROLES           AGE   VERSION
imaster00   Ready    control-plane   64m   v1.30.5

Now that networking is fully functional, we can see the initial node is in a ready state.

More control plane nodes

Now that we have configured networking, we can begin adding more nodes. We would start by adding more control plane nodes by using the kubeadm join command that includes control-plane and certificate-key parameters. We are not going to be adding additional control plane nodes in this post.

Worker Nodes

sudo kubeadm join xd7control:6443 --token ui1vd6.kvea6irgh5ulrka1 \
	--discovery-token-ca-cert-hash sha256:b9d49ec4a75cf7457e39f49d974e93a9aef6c6cec6c2bfede92c3b7c8223f221

On each worker node, we’ll run the kubeadm join command.

ubuntu@imaster00:~$ kubectl get nodes -o wide
NAME        STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
imaster00   Ready    control-plane   68m   v1.30.5   10.0.0.12   <none>        Ubuntu 24.04 LTS   6.8.0-31-generic   containerd://1.7.22
inode00     Ready    <none>          98s   v1.30.5   10.0.0.13   <none>        Ubuntu 24.04 LTS   6.8.0-31-generic   containerd://1.7.22
inode01     Ready    <none>          88s   v1.30.5   10.0.0.14   <none>        Ubuntu 24.04 LTS   6.8.0-31-generic   containerd://1.7.22

And, after watching the pods intialize and come online, we can check our nodes again and see everything ready.

Github Issues / Discussions - available for suggestions/corrections

Create a new instance with an external ip address or that has external ports mapped for 22, 80 and 443. Create an internal ip address on the same or a second interface. Initialize the instance to include privoxy and nginx using cloud-init. Now we provision the host in the cluster so we can get our initial services online. (This also establishes the internal ip address if you don’t choose it in advance)

SSH config

We’re going to be making changes to the local ~/.ssh/config file to facilite communicating to our cluster.

Host xd7tower
    HostName 162.200.24.101
    User ubuntu
    IdentityFile ~/.ssh/id_rsa_blog_skillcadet

Now we ssh to the tower machine for the first time:

ssh xd7tower
The authenticity of host '162.200.24.101 (162.200.24.101)' can't be established.
ECDSA key fingerprint is SHA256:k6AjX1v7wDxj/QYffwiHdAKS8rR2cGIMTIQEben5UH8.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '162.200.24.101' (ECDSA) to the list of known hosts.

We’ll need to approve the host fingerprint, which we often need to do blindly unless we have a secure console to the node/instance.

Cluster provisioning

Now, when we spin up our hosts, we’ll be able to configure ssh to reach the hosts through the bastion server.

How many nodes do we spin up? This series of blogs will walk through setting up a cluster with stacked control plane nodes (vs using an external etcd cluster), which means we can select an odd number of control nodes. One control node is obviously necessary for the cluster to start. For high availability, we would want 3 control nodes, so that the cluster will continue to operate smoothly even if (intentionally or otherwise) we have one control plane node down.

Host xd7imaster00
    HostName 10.7.222.13
    User ubuntu
    IdentityFile ~/.ssh/id_rsa_blog_skillcadet
    ProxyJump xd7tower

Host xd7inode00
    HostName 10.7.222.12
    User ubuntu
    IdentityFile ~/.ssh/id_rsa_blog_skillcadet
    ProxyJump xd7tower

Host xd7inode01
    HostName 10.7.222.14
    User ubuntu
    IdentityFile ~/.ssh/id_rsa_blog_skillcadet
    ProxyJump xd7tower

Finally, we add our internal hosts to the ssh config. In addition to the Hostname information, we include the ProxyJump line so the ssh knows we reach the internal hosts via sshing through the tower host.

$ ssh xd7imaster00
The authenticity of host '10.7.222.13 (<no hostip for proxy command>)' can't be established.
ECDSA key fingerprint is SHA256:6lNvogJRmjYR9n1ZJs6KrCAXnGvsDmbW6RO0SG9DPvE.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.7.222.13' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-31-generic x86_64)

ubuntu@imaster00:~$

Now, we should be able to ssh directly to the internal hosts.

Freshness - this, and all, k8s tutorial suffer from staleness quite quickly. In general, tutorials have been succesfully applied across versions, however this changes slightly with the new (2024) release structure of community supported kubernetes, where each version has it’s own distribution url. The GPG keys, SSH keys, and apt sources that will change will be highlighted in the document.

This version, dated October 1st, 2024, uses:

Why this post

There are plenty of tutorials out there that walk through manually setting up a kubernetes cluster, however, in the real world, we want to make things repeatable and not tedius. There are a few ways to provide this capability, most of them, like Anisible/Puppet/Chef/Salt/Terraform/Cloudformation all require additional infrastructure.

But, what if there was a way to provide repeatable configuration, across cloud and virtualization platform providers, using a text file that can be checked into a code repository?

cloud-init

Let me introduce you to Cloud-Init. Using cloud-init, we will perform all the basic installation steps to bring a stock system image and prepare it to receive a kubernetes node.

We’ll still have plenty to do to bring up and manage the cluster, however, we can start at the fun part and focus on what makes the nodes different.

Let’s take a look at the cloud-init.yaml that we’ll be crafting.

preamble, disable swap and (optional) customizable hostname

#cloud-config
hostname: inode01
create_hostname_file: true
mounts:
- [ swap ]

We start with the #cloud-config stanza. Next we have an choice. We can create separate cloud-config.yaml files for each node and provide our own hostname, or, depending on your provider, we leave the hostname/create_hostname_file stanzas out and your node will be called something random or will be called ubuntu by default. The hostname stanza is the only custom resource and if your hosts are called ubuntu, it’s simple to change manually when you login to the host the first time.

Finally, we make sure that swap is disabled by providing no options in the config. Later, we’ll run a command to make sure swap is currently off.

Add admin user

users:
- name: ubuntu
  gecos: Ubuntu User
  shell: /bin/bash
  groups: users, admin
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys:
    - ssh-rsa AAAAB3Nza......= troy@skillcadet

Next, we want to address the admin user. We want to disable root ssh login, so we need a regular user with admin privileges. We also want to secure our communication channel with a public/private key instead of using passwords for login.

  hashed_passwd: "$6$rounds=4096$/NM2KaSQ595tb74s$38kIiWZXh4MMpb95/0TdGAm.Wr7J9SMFnb.CSsxlsU0oE7pgkGcCs2JpJHDCuGXMNvVEmpU01SBONYmVxpAv8/"
  lock_passwd: false
  sudo: ALL=(ALL:ALL) ALL

You can, alternatively, replace the sudo stanza with the above triplet to enable password protected sudo. This isn’t really ideal either, as providing a hash, even as strong as this, is very crackable if discovered, using a longer complex password could help. You can generate a hashed password with the following:

mkpasswd -m sha-512 -R 4096

Create system files

write_files:
- path: /etc/sysctl.d/kubernetes.conf
  owner: root:root
  permissions: '644'
  content: |
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.ipv4.ip_forward = 1
- path: /etc/modules-load.d/k8s.conf
  owner: root:root
  permissions: '644'
  content: |
    overlay
    br_netfilter

Additionally, we need a few configuration files to be written, luckly that is easily accomplished with cloud-init.

update config changes

In the home stretch, we encounter the parts of the config that change.

We’ll add these into the correct locations below:

Add apt sources

apt:
  sources:
    containerd.list:
      source: deb https://download.docker.com/linux/ubuntu $RELEASE stable
      key: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----

        mQINBFit2ioBEADhWpZ8/wvZ6hUTiXOwQHXMAlaFHcPH9hAtr4F1y2+OYdbtMuth
        lqqwp028AqyY+PRfVMtSYMbjuQuu5byyKR01BbqYhuS3jtqQmljZ/bJvXqnmiVXh
            SNIPPED
        =0YYh
        -----END PGP PUBLIC KEY BLOCK-----
    k8s.list:
      source: deb https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /
      key: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        Version: GnuPG v2.0.15 (GNU/Linux)

        mQENBGMHoXcBCADukGOEQyleViOgtkMVa7hKifP6POCTh+98xNW4TfHK/nBJN2sm
        u4XaiUmtB9UuGt9jl8VxQg4hOMRf40coIwHsNwtSrc2R9v5Kgpvcv537QVIigVHH
             SNIPPED
        oyA0MELL0JQzEinixqxpZ1taOmVR/8pQVrqstqwqsp3RABaeZ80JbigUC29zJUVf
        =F4EX
        -----END PGP PUBLIC KEY BLOCK-----

Here, we introduce the two apt sources so we can retrieve the necessary packages.

There are alternate ways to specify apt sources instead of including the PGP keys directly.

Add required packages

packages:
  - curl
  - gnupg2
  - software-properties-common
  - apt-transport-https
  - ca-certificates
  - containerd.io
  - kubeadm
  - kubelet
  - kubectl
  - fail2ban

Fail2ban

All the packages above are consistent with most tutorials, with the expection of fail2ban. This is added since we need to have an SSH port open to manage our system.

Now, we’ve already disabled root login and disabled password logins in favor of SSH keypairs, however to make things even more secure, Fail2ban will monitor logs for ssh login failures (and other things if we canfigure fail2ban do so) and will block access to the SSH port for repeated offenders. The block goes away after a few minutes, so if you mess up a bunch of times in a row, just be patient. This wait period can frustrate a user, but it makes it impossible to brute force a login.

Configure containerd, lock packages and start fail2ban

runcmd:
  - [ swapoff, -a ]
  - [ modprobe, overlay ]
  - [ modprobe, br_netfilter ]
  - [ sysctl, --system ]
  - containerd config default | tee /etc/containerd/config.toml >/dev/null 2>&1
  - [ sed, -i, 's/SystemdCgroup \= false/SystemdCgroup \= true/g', /etc/containerd/config.toml ]
  - [ systemctl, restart, containerd ]
  - [ apt-mark, hold, kubelet, kubeadm, kubectl ]
  - [ systemctl, enable, fail2ban ]
  - [ systemctl, start, fail2ban ]

Almost finished now, we execute some commands to prepare the system.

First, we apply system changes we made with write_files above. Then, we need to generate a containerd config file so we can override the default SystemdCgroup setting with ‘sed’. Next, we restart containerd to use the proper Cgroup. We follow that with marking the kubernetes packages as HOLD so they do not get upgraded automatically. And then we enable and start fail2ban.

Restart to start with a clean environment

power_state:
  mode: reboot
  message: First Boot Complete

Finally, we reboot to clear things up and start with a fresh boot, ready for us to login and create the cluster.

Next up, my tutorials on manually bringing up a cluster from this point and on connecting an external load balancer to a cluster nodeport.

Github Issues / Discussions - available for suggestions/corrections