Then you need 9. A simple browser based text sharing tool.

It’s anonyous. You don’t need to register.

It’s secure. Once a connection is established, the text is encrypted and sent directly to the other device. No server in between.

It’s quick. Quickly scan a QR code if you use your mobile phone. Or type in the 6-digit code if you use another computer (which shortly utilizes Cloudeflare’s WebRTC implementation to establish a direct connection).

If your devices are not on the same network 9 utilizes STUN by Google to establish a direct connection. Still, the text is being exchanged directly between the two devices.

GitHub Repository

nine online

We just got our hands on an internal Anthropic strategy slide, and it will completely blew your mind. 🤯

Here is an absolute masterclass in margin optimization. 👇

The strategy is brutally simple:

1️⃣ Silently degrade the model to ruthlessly cut inference costs. 📉 2️⃣ Wait for the market to complain. 🗣️ 3️⃣ Re-release the ORIGINAL, un-nerfed model for 2x the price under a slick new name. 🚀💰

Let that sink in. 🧠

They aren’t actually training better models. They are just silently routing your queries to heavily quantized, smaller versions of the current one.

Then? They charge you DOUBLE for a “next-gen” release that merely restores the full precision weights. 💸🔥

This is the ultimate SaaS infinite money glitch. 🤑 A true game-changer in AI monetization.

Agree? Let me know your thoughts in the comments! 👇

/s

Yeah, I know, there are many many confidence calculators out there. But this one is different, because it’s mine ‘(͡• ͜ʖ ͡•)’

No seriously. All the other options out there are either a little too complicated, lacking features or missing simple layman explanations - because: I am not a data scientist, a market researcher or a rocket scientist. I’m just…a dude, I guess.

However: I built this one as an Excel file, because I was working with Adobe Target for a huge… very huge… very infamous - I mean - famous of course - German railway company. And later improved it to look even nicer. And then I got the hint: Make it a product. Yeah well, here it is. It’s free. Thank me later. (⌐■_■)

Here it is:

• you can easily add up to 5 variants
• you can easily configure confidence levels or other parameters and they are explained in a simple but hopefully sufficient way - I will also repeat it here - for your convenience
• you can share the results - always, everywhere, because the parameters are hidden in almost plain sight in the URL
• you can see two charts showing you, when the test will reach a certain confidence level, like how many days you have to wait
• or how different traffic numbers will affect the confidence level

confidence.nickyreinert.de

Dont hack the P!

Well, now you might think: I have an interface. I will just update it everyday and as soon as I reach the required confidence level, the test is succesful.

That’s called p-hacking. Don’t do that. What if the very next day, the numbers are going down again, because this one special day external and unknown factors influenced the test?

That’s why you plan a test and define a target* to reach - not only the confidence level, but also the running time and the number of visitors.

Some basic information about A/B testing

The hypothesis

This is where everything starts: Your assumption. This is mostly a bookkeeping task and does not affect the calculation itself.

The null hypothesis says: “The test will have no effect on the conversion rate.” Your goal is to disprove it.

The alternative hypothesis says: “The test will have an effect on the conversion rate.” Your goal is to find enough evidence to reject the null hypothesis in its favour.

Type-I and Type-II errors

The Type-I error means the result is a false positive: The test suggests an effect exists — while in reality, no effect exists. You want to avoid that, too.

The Type-II error means the result is a false negative: The test fails to detect an effect — while in reality, a real effect exists. You want to avoid that, apparently.

Test Type

It sounds strange, but it’s easy: A test can have two effects: A conversion rate can either increase or decrease.

A one-sided or one-tailed test assumes that only one direction is possible. A two-sided or two-tailed test assumes that both directions are possible. A one-tailed test requires less data and reaches confidence faster, but there is a risk: What if the effect is in the other direction? You won’t see it because it’s not part of the calculation. You want to see the calculation?

$$ p_{\text{one-tailed}} = 1 - \Phi(z) \qquad\qquad p_{\text{two-tailed}} = 2,\bigl(1 - \Phi(|z|)\bigr) $$

What’s the confidence level?

There are many mathematical and very complex ways to explain that, but actually, what I think is important to know, is this:

You want to make sure that the results of your A/B test are not just coincidence or luck. The p-value estimates how compatible your observed data is with the assumption that there is no real effect — in other words, how likely it is that what you see is just noise.

Before running a test, you typically choose a significance level — for example 5%, corresponding to a 95% confidence threshold. In this calculator, confidence is reported as:

$$ \text{Confidence} = 1 - \text{p-value} $$

Strictly speaking, confidence levels and p-values are distinct statistical concepts. But this representation is commonly used in A/B testing tools because it is intuitive for non-specialists — and it is what your calculator reports.

The higher the confidence level, the lower the chance of a Type-I error.

What’s the significance level?

In simple words: It’s the confidence level but the other way around, like:

$$ \alpha = 1 − \text{Confidence} $$

It’s closely related to the p-value, like: The significance level is the threshold, the p-value is the result.

The lower the significance level, the lower the chance of a Type-I error.

What’s the power of the test?

This is probably the second most important metric of a test and it’s almost the opposite of the confidence level. Confidence tells you how sure you are the effect isn’t noise; power tells you how likely the test is to detect a real effect if one exists.

Assuming there is a real effect, how likely is this test to detect it?

Power is not part of the p-value calculation. Instead, it helps you estimate before the test starts how many visitors are needed to reliably detect an effect.

$$ \text{Power} = 1 - \beta $$

What’s beta?

While $\alpha$ and $\text{Confidence}$ are about the Type-I error, $\beta$ is the probability of a Type-II error. The calculation is a little more complex. It requires not only your test variables but also constants such as the z-score.

The lower the $\beta$, the lower the chance of a Type-II error. The higher the power, the lower the chance of a Type-II error.

Uplift, Visitors, Conversion Rate and Distribution

They are quite simple metrics, as they are just common “performance indicators”. Visitors are the people you reach. Conversion can be quite anything that you want to measure and improve. The conversion rate is the percentage of visitors that convert. And the uplift is, either the expected or observed change in the conversion rate.

The other parameters

There is a complex mathematical pipeline behind the calculation of the confidence level for each variant. It starts with the pooled conversion rate:

$$\bar{p} = \frac{c_1 + c_2}{n_1 + n_2}$$

Where $c_1$ and $c_2$ are the number of conversions for variants 1 and 2, and $n_1$ and $n_2$ are the number of visitors for variants 1 and 2.

From this, the standard error SE is calculated:

$$SE = \sqrt{\bar{p} \cdot (1 - \bar{p}) \cdot \left(\frac{1}{n_1} + \frac{1}{n_2}\right)}$$

Which goes into the calculation of the z-score using $p_1$ and $p_2$, the conversion rates of variants 1 and 2:

$$Z = \frac{p_2 - p_1}{SE}$$

Followed by the calculation of the p-value using the z-score and the cumulative distribution function $\Phi$ (shown here as one-tailed; for a two-tailed test it’s $2(1 - \Phi(|Z|))$):

$$p = 1 - \Phi(Z)$$

The what? The cumulative distribution function $\Phi$! Historically it came from lookup tables — today it’s calculated directly by software. You don’t want to know more. (◔_◔)

The p-value is the first important metric, it must be below the significance level $\alpha$ to reject the null hypothesis.

$$\text{Confidence} = 1 - p$$

Therefore, the reported confidence must exceed your chosen confidence threshold before you reject the null hypothesis.

But What Is the Bonferroni Correction?

If you have multiple variants compared against one control group, you may want to apply the Bonferroni correction. It’s meant to reduce the chance of a Type-I error.

How does it work? You simply divide the significance level $\alpha$ by the number of comparisons $m$. So if you have 3 variants, you would divide $\alpha$ by 3. The adjusted significance level is then used instead of the original one when evaluating each variant’s result — raising the bar for everyone. The idea is: if you test many variants at once, the probability of seeing at least one seemingly significant result purely by chance increases. I mean - there are so many colors!

For many people a Google account is the central access point for their digital identity. For a great many people, in fact. Through SSO, we gain access to many platforms and systems. Communication often runs entirely through Gmail. That starts with forum logins, continues through GitHub accounts, and can extend all the way to tax returns or communication with your health insurer. It may even lock you out of your phone, that you use to pay or log in to your banking account. Google Mail is way more than communciation. It’s access and identity. The digital sphere has completely absorbed real life. More for some, less for others.

That is reason enough to keep advocating for digital sovereignty. But that is not the point here. I want to raise a different question and illustrate the problem with a current report: Google has suspended the account of a Japanese manga artist [ reddit.com ]. This is not the first time and it does not only concern Google. Microsoft is also known for deleting or suspending accounts for different reasons:

• ‘Microsoft killed my online life,’ Microsoft is reportedly banning Palestinians in the U.S. for life for calling relatives in Gaza
• Need help, locked out of Google account with 10 years of personal data
• Microsoft Account Suspension Leads to Loss of 30 Years of Personal and Professional Data
• Account Terminated without Warning: Victim of Scam - Seeking a Second Chance after 1.5 Years

At first, that sounds trivial. A few emails are missing, probably some images, that is all? Where’s your BaCkUp?

Well… as I described at the beginning, the simple email account, whether from Google or another provider, has evolved from a purely digital communication medium into a central link to real-world identity. It is not just about emails, but about the representation of one’s own personality in digital space.

That gives providers like Google a special responsibility that goes far beyond the customer-company relationship.

Even if Google acts primarily, and understandably, according to economic interests, its influence on an essential basic service is enormous. When it comes to access to the internet, policymakers have imposed important rules on providers, going so far as to recognize that access as a basic right. Yet major identity platforms like Google remain largely unregulated when it comes to responsibility in the context of public provision. As a private company, Google can decide on its own, and relatively opaquely, when an account is deleted. There is no neutral appeals process or even an “in dubio pro reo” rule.

That is a problem. The call for digital sovereignty is not the right step here either, because it shifts responsibility onto consumers. Just ask your parents, who are in their mid-60s, why they not host their own email and LDAP server. We should not overlook the fact that not everyone is readily able to implement sovereign alternatives.

That is why politics is needed: to better oversee the rules for account management by large companies, so that decision-making power is placed in the hands of a neutral party.

A few weeks ago I dived into a new “hobby”: remote-controlled semi‑pro vehicles — a birthday present for my child.

So I started researching. For me, the priorities were robustness and modularity. I wanted my kid to be able to drive through rough terrain and I knew that parts would occasionally break. That meant spare‑part availability had to be good.

At some point I found the Arrma Gorgon, a 1/10 monster truck. Retail price without battery or charger: €180. Not cheap, but reasonable. Since driving together is twice the fun, I bought two vehicles. I also ordered four LiPo batteries. (This is where the whole compatibility madness begins: LiPos need a dedicated charger.) Because the stock model uses brushed motors, I also bought two fans so I wouldn’t be forced to stop after 20 minutes of driving. And the fans came without screws, so I added a 500‑piece RC screw kit to the cart. It adds up — but there’s more.

More issues followed. For example, the fan connector didn’t fit the controller, so out came the side cutters and I trimmed the protruding plastic. Great start for “modularity.”

Here’s my initial shopping list:

The first day was a blast — the electric motors have plenty of power and we had an enormous amount of fun. But already on day one the first disappointment: a front suspension rod broke.

I temporarily repaired the shock using a suitable piece of hose coupling until the replacement shock set arrived — fixed professionally with gaffer tape, naturally.

Costs for that repair:

The next day brought the next surprise: within a few hours the steering failed. At first it’s confusing since it’s not immediately clear what’s broken. Eventually you find the steering servo — and it failed on day two, on both vehicles.

From my perspective the issue is clear: the shock protects the body but not the tires. The tires are simple, tubeless, very soft rubber. Every collision transfers force directly to the control arms, the steering and the suspension. The servos use cheap plastic gears that don’t last. That’s a design flaw — or am I the only one noticing?

So I grabbed the craft scissors, cut a piece of HD‑PE pipe and screwed it to the small shock.

I ordered a new pair of servos.

Costs for the servos:

The installation is quite fiddly because the control arms sit under a plastic brace. Eventually it was fixed and we went out again…

But then the next problem surfaced:

Less than a week later the drive motors failed. Troubleshooting was tricky because it’s not obvious whether the battery, controller or motor itself is at fault.

Eventually it became clear: both motors were defective, despite the added cooling. I contacted support, which I probably should have done earlier. The dilemma: I wanted to keep the vehicles because my kid had already grown attached to them.

Support was friendly and quickly offered replacements for both motors. Not wanting a fight, I ignored the broken shocks and servos and accepted the replacement motors — I just wanted to drive.

When the replacements arrived I installed them and everything initially seemed fine. After a few minutes of driving smoke rose from one truck. This time it was user error: in my excitement I forgot to reconnect the motor/controller insulating sleeve — short circuit.

And a short circuit takes several components with it because there appears to be no fuse.

Okay, I told myself, I wanted to upgrade to brushless motors anyway. So the research machine started again. One tip: don’t blindly trust Gemini or ChatGPT — they often recommend things that simply don’t fit. Compatibility matters and the AI often misses that.

I found what looked like a perfect brushless combo, but a motor doesn’t come alone: you need a matching ESC and typically a separate receiver. The stock controller/receiver is mounted with screws — the new receiver had no mounting points.

Anyway, the new drive combo arrived and I tested it on the bench: controller + motor + battery.

The battery? It had an IC3 connector — plus and minus plus a small pin for the charger to query capacity/status. The ESC had an EC3 connector. They don’t match.

So out came the adapters.

Adapter arrived. Adapter connected. ESC connected. Battery connected. Motor: runs.

So far, so good — until I tried to mount the motor. The stock motor routes its wires out the back; the new motor has a side exit and is shorter. In short: it doesn’t fit. Frustration ensued.

Costs for the brushless combo that I ended up returning:

I emailed the seller and returned the motor (shipping at my cost). Research continued: the motor needed to be longer or have rear‑exit wires. I found another candidate:

Ordered, waited, received — with a tiny spark of hope.

The longer motor fit the case exactly — great — but then the pinion issue appeared: the longer motor’s shaft is thicker (5 mm) while the stock pinion has a 3.175 mm bore.

More searching led me to a suitable pinion on Amazon. Two days later it arrived and looked perfect for the 5 mm shaft.

But it didn’t fit. The teeth barely meshed with the gearbox gear and the diameter was too large, causing excessive friction. The plastic gearbox gear wouldn’t take the load for long.

Rage. Despair. Doubt. The hunt continued: the right pinion should have no more than 12 teeth, module 0.8, and a Safe‑D (D‑shaped) bore to prevent slipping. I skipped the Safe‑D requirement and hoped a plain 5 mm bore pinion with 12 teeth and 0.8 module would work:

That pinion worked. The drivetrain was back together, but this is a jury‑rigged long‑term provisional solution. The grub screw that came with the pinion was too long (WTF), so you must be careful not to push the pinion on too far. The IC3→EC3 adapter also eats space in the battery bay, making battery swaps fiddly. Finally, you need to secure receiver, ESC and switch somewhere in the chassis — there are mounting holes, but you need a kind of cage. For now I used Velcro and double‑sided tape. The result is a chaotic setup far from the modularity I hoped for.

Conclusion

First — self‑criticism: the short circuit was my fault. That was spectacularly stupid.

༼ʘ̚ ʖ ʘ̚༽

That said, once you know what to look for the technical specs aren’t that hard to understand. You just have to know where to look. A quick note: I initially tried to use Gemini, ChatGPT and Perplexity to find the right parts. The AIs repeatedly recommended unsuitable parts because they didn’t take compatibility details into account.

Even specific follow‑up questions often led to wrong links or bad recommendations, so I paid a lot of tuition. My tip: talk to the vendor directly if possible — don’t rely exclusively on AI for specific hardware compatibility.

At least one thing came out of it: tinkering is fun. For anyone interested in swapping the motor on an Arrma Gorgon, here are the essentials — and many points apply to other vehicles too:

• There are many battery types. LiPo is the de facto standard; each battery class needs a compatible charger. Check whether the battery has a telemetry pin on the connector.
• The motor must be long enough if the wires exit the side — aim for at least 60 mm length; too long is also a problem. Short motors typically route wires out the rear.
• You usually need a new ESC and possibly a separate receiver. Some combos include everything, some don’t.
• Pay attention to connector types: EC3, IC3, XT60, male/female — they matter.
• The stock pinion has a 3.175 mm bore — check the motor shaft diameter and the pinion module (tooth size).
• The original shaft often has a D‑shape (Safe‑D) to prevent slipping; a simple round shaft requires a suitable pinion and grub screw.
• The number of teeth and the module must match the gearbox gear; changing pinions changes the drive ratio.
• If you add a fan you may need longer screws.
• Make sure the new motor can be secured to the chassis — some mounts require custom brackets or Velcro.

Final note on quality and design: the stock brushed motor is exposed near the drive shaft and is vulnerable to dust ingress via the ventilation slots. That is not ideal for long‑term off‑road use. A closed brushless motor is a sensible upgrade if you plan to run the truck in rough conditions.

The decision to use the long front bumper is debatable — while it protects the chassis it also transfers impact forces to the control arms and servos, increasing the risk of servo failures. The long control arms act as levers that strain the servo’s plastic gears.

In my view the Arrma Gorgon in stock trim is not designed for hard off‑road use — despite what the promo videos suggest.

https://www.arrma-rc.com/de/product/1-10-gorgon-2wd-rtr-brushed-monster-truck-blue/ARA3230T1.html

But maybe Arrma has an explanation?

[1] https://www.valuize.co/customer-value-equation/ [2] https://count.co/metric/peak-support-hours-analysis [3] https://thenextweb.com/news/anthropic-800-billion-valuation-revenue-30-billion-ipo [4] https://www.saastr.com/anthropic-only-has-5000-employees-almost-no-one-has-ever-been-this-efficient-thats-by-choice/ [5] https://www.pymnts.com/artificial-intelligence-2/2026/anthropics-claude-draws-in-record-number-of-consumers/ [6] https://mlq.ai/news/anthropics-claude-experiences-record-surge-in-paid-subscriber-growth-throughout-2026/ [7] https://www.getpanto.ai/blog/anthropic-ai-statistics [8] https://www.citadelsecurities.com/news-and-insights/2026-global-intelligence-crisis/

A (mostly) browser based toolbox for a couple of common, simple image editing tasks:

• resizing, cropping and color adjustments
• pattern generation with a couple of different algorithms
• adding annotations like text or shapes
• showing common social media formats
• OCR extraction with tesseract, in the browser
• simple animations like Ken Burns
• ASCII-converter
• steganography
• and of course EXIF data viewer, reverse image search, etc.

So there’s a lot to explore here. Some functions require a backend, like I built in an “AI image identifier”. Also the user has (very limited) access to some free external image databases, like picsum.photos or unsplash.com . As CORS would block requests like that, I implemented a server side proxy for that.

As this post is gaining more attention than expected on HN, I want to make some things clear - although I thought they are obvious.

It sounds like a rant about Claude’s quality in general - if you don’t read to the end. But my concerns are more focused on the support performance and token issues - fully aware of the challenges a company this size faces and assuming the people at Anthropic are working hard to make things better. I am pointing at some “bad design decisions” they probably made. The “quality issues” are just the cherry on top of the cake.

After all, Claude Code is delivering and I use it to build stuff. Still, I experienced a degradation in quality. It just takes longer. Which is a relative observation. However I know that this is highly subjective, too. That’s what that comment in a paragraph means: “The failure usually appears in front of the screen”. Of course the agent is only as good as the operator and their instructions.

I am coding for a couple of decades now, I like to get my hands dirty. Three years ago I implemented AI into my workflow. It started with code completion and now I am at the point where I barely write code. For me “software engineeering” is not constituted by the simple act of writing code. It’s about conducting tools, being creative, understanding the problem and delivering a solution. Still - and that what this comment in another paragraph means: “While I was browsing the model’s thinking log - which I strongly suggest doing not only occasionally” - I am escorting the agent while it’s working. I still have to figure out a concept, think about a data model and verify it’s implementation. That’s what software engineering is about. It’s not n LOC…

Having said that - enjoy this post and stay happy, fellow developers.

First enthusiasm

A couple of weeks ago I subscribed to Claude Code, and during the first few weeks I had a really nice experience. It was fast, the token allowance was fair, and the quality was good.

I learned they had raised the token allowance for non-rush hours , and since they opposed some governmental rules, it felt good to support the right cause.

(づ ￣ ³￣)づ

However… for about three weeks now my initial enthusiasm has been rapidly waning.

Poor support

It began with an issue three weeks ago. I started working in the morning after about a ten-hour break; enough time for my tokens to refresh.

I sent two small questions to Claude Haiku. They were simple questions, not even related to the repository.

Suddenly, token usage spiked to 100%.

Have a nice break…

I contacted their “AI support bot”, which returned some default support nonsense and didn’t really understand the problem. So I asked for human support. A couple of days later a - what appeared to be - human support person sent a reply. It began like this:

“Our systems are detecting your inquiry is regarding usage limits on your Pro or Max plan.”

Yeah, well — it’s the Pro plan. Seems like your systems weren’t actually queried; it was just a default intro and probably a default answer, because:

This was followed by an extensive what seems to be copy-and-paste answer from their docs explaining how daily and weekly limits work.

And it closed with the typically frustrating line, that no customer likes to read at the end of an e-mail and which is just the classical middle-finger of customer support - we don’t care if your problem is solved or not, we declared it closed.

“Note that further replies to this ticket may not be monitored. If your request is not regarding usage limits on your Pro or Max plan, or you need additional support, please visit our help page at”

Great! Sending an automated e-mail that does not refer to the actual problem and then closing the channel. Thanks for nothing, I guess? Or was I wrong. I asked Claude Haiku:

@Haiku:

See the customer’s request here and the response from the AI and later W***** - did they answer the concern/question of the customer?

(╯°_°）╯︵ ┻━┻

Declining quality

In the following days and weeks, the quality was far from satisfying my needs or matching my initial experience. While I used to be able to work on up to three projects at once, now the token limit was exhausted after two hours on a single project.

And the quality was degrading. I am fully aware this is quite subjective and that the quality of the agent is always heavily impacted by the operator. The failure usually appears in front of the screen. But hey, I also develop using Github’s Copilot, OpenAI’s Codex and I am running my own inference with OMLX and Continue using Qwen3.5-9B. I’m not the expert, I’m lazy sometimes but I probably know a thing or two.

Let me give you this wonderful example: yesterday I asked Claude Opus to refactor a project.

While I was browsing the model’s thinking log - which I strongly suggest doing not only occasionally - I found this:

Rather than editing every slider in JSX, I’ll add a generic initializer in ui-events.js that auto-injects value displays for all range inputs that lack one.

This is clearly bad practice. It’s a cheap workaround you wouldn’t expect even from a junior dev; it reads like someone who just doesn’t want to deliver a good result. My response:

“you can’t be serious — is this how you fix things? just WORKAROUNDS????”

At least Opus admitted:

“You’re right, that was lazy. Let me do it properly — add the labels directly in the JSX and wire them explicitly.”

Needless to say, this shortcut cost me around 50% of my five-hour token allowance.

(ง •̀_•́)ง

And even more…

Now this cache topic comes up - among others . at least they are talking about it openly. The problem was: when you get back to work after some time, your conversation cache is gone and the model starts reading your codebase again. Cost-wise this is smart. But experience-wise? It means you paid tokens for the initial load and, after a forced break because the five-hour token window hit its limit, you pay again for the same load.

Think that’s all? Wait, I also got this funny anecdote: all of a sudden the weekly window changed from today to Monday. OK, I was thankful because it came with a reset to zero. But still: what is going on, Anthropic? Not only that — while I was working on my project, watching token usage with Argus-eyed vigilance, this little warning popped up:

Wait, what? I’m neither part of an organization nor do I see any hint why I suddenly have to worry about a “monthly usage limit” — also the hourly and weekly limits were still not exceeded. What is happening right now?

Turns out — two hours later - it allowed me to continue working. The warning was gone.

At least this documentation does not mention a monthly usage limit. And the settings page only lists the limits for the current session and week.

So… what is this monthly limit all about, Anthropic?

Sorry to let you down, Anthropic

I am a huge fan of the product. Theoretically everything just works like a charm; it offers so many opportunities. I built my own harness around Claude , I admire Claude Caude who work’s in the background on a bunch of GitHub issues, I love using Claude Cowork to continue writing my Nerd Enzyklopädie . So many thoughful features.

I increased my productivity by an order of magnitude, and it’s really thrilling to see how these trillions of ideas crawling through my head are now only a blink away - easier and quicker to realize than four years ago.

And I understand the technical and organizational challenges when offering a product like that. It’s not easy to benefit from scaling effects when you sell inference. Every additional hour and every new customer requires the same amount of compute. That’s the curse of incremental costs in this line of business.

But…

…it seems like Anthropic cannot handle too many new customers at once, so I took that load off Anthropic and cancelled my account.

(ʘ‿ʘ)╯

Last night, around 3 am, I woke up and thought: Wouldn’t it be nice to have an animated ticker in Excel? Like text that scrolls from right to left? With just one big formula? Welcome to the second episode of my series “How to create weird algorithms” First one was about creating a maze in JavaScript ). In this article we will create a nested Excel formula to solve a problem that probably not knows exists. Until now.

TYL — Today You’ll Learn

• How to build complex nested formulas from the scratch
• The MAP-formula to create an array and modify it’s values
• The LET-formula to declare variables to clean up nested formulas
• How self-referencing cells in Excel could be useful
• How to use INDIRECT and R/C referencing

Preparations

First let me introduce you into an widely ignored feature in Excel: Iterative calculations. Excel tries to protect you from “circular references” and therefore disables this feature by default.

Our ticker requires a formular that references itself. So you need to enable “iterative calculations” in the settings and set “maximal iterations” to 1 :

Now let’s get back to your worksheet and start from the scratch:

How to draw a little canvas from just one cell?

In our case we need a one-dimensional array to use as a canvas. But how to create an array over multiple cells, just using one formula?

Well, there are a couple of ways to do that, like SEQUENCE, MAKEARRAY or INDIRECT (and probably more). For our problem we will use MAP and SEQUENCE. MAP allows us to create an array and modify the values within this array. Let’s try a simple example:

1=MAP({1;2;3;4;5;6;7;8;9;10};LAMBDA(var;var))

This will put the numbers from 1 to 10 to your worksheet. You could even add additional rows. For our case the one dimensional array is sufficient. The LAMBDA function allows you to modify the values. To make it more handy, let’s add SEQUENCE to construct the array and LET to define variables:

1=LET(
2    length; 10; 
3    MAP(
4        SEQUENCE(1;length);
5        LAMBDA(val;val)
6    )
7)

LET is a good way to declare variables that you may need in your nested formula.

How to simulate an “animation”?

We’re not really going to animate something. We’re just going to create the impression of animated text. To achieve this we first need to add some momentum. That’s were rely on the self-referencing feature. We just add the value of the first cell to all the other cells:

1=LET(
2    inc; INDIRECT("R" & ROW() & "C" & COL(); 0);
3    length; 10; 
4    MAP(
5        SEQUENCE(1;length);
6        LAMBDA(val;val + inc)
7    )
8)

OK, this probably looks weird, so I’m gonna explain it. In the second row we are assigning a calculation to a variable called inc, this will be our “incrementor”. Instead of a fixed reference like A1 we use an indirect reference. This makes it easier to use the final result everywhere we want. INDIRECT takes the value from the current row and current column: This is the 1 from the sequence. The second parameter of INDIRECT is a 0. This enables the R/C reference: referencing a cell by it’s row and column index instead of the “common” A1 notation.

Now as we have this value from the first cell, we just add it to every cell of this range — which results in a row of incrementing numbers. Press and hold F9 and see what happens:

From now on, when I say “let it run”, you need to press and hold F9 and see the result.

Now let’s add the text. We want to display “hello, world” and therefore provide this as a variable named ticker:

 1=LET(
 2    ticker; "hello world";
 3    length; LEN(ticker);
 4    inc; IF(
 5        INDIRECT("Z" & ROW() & "S" & COL(); 0) > length;
 6        1;
 7        INDIRECT("Z" & ROW() & "S" & COL(); 0) 
 8    );
 9    MAP(
10        SEQUENCE(1; length);
11        LAMBDA(val;
12        IF(val = 1; 
13            val +  inc;
14            MID(" " & ticker; val + inc; 1)
15        )
16    )
17)

The ticker-declaration should be clear as well as the length-dependency. The length was set to 10 before, now it’s the length of the ticker-text.

What looks a bit complicated is the improved declaration of the incrementing variable (inc): Actually we just check if the variable is greater than the amount of available letters. If so, reset it to 1 to start over.

Before we look at the LAMBDA let’s be clear about the “canvas”: We need one cell that contains an incrementing value — the first cell. All the other cells are meant for “rendering”.

The first IF-condition checks if the value of the particular cell equals 1. This would be our counting cell and that’s why we increase only this cell’s value.

The MID-formula takes care of the rendering-part: The second cell in our one-dimensional array should start with the H, the first letter of our ticker. The third cell displays the second letter E and so on.

You see the discrepancy? The second cell holds the value 2, but we need the first letter. We are one counter ahead, that’s why we add the empty char before the ticker text.

What happens if your run it?

Well, we have a pretty good loop. It’s not that smooth, but we get to that in a second.

Finish the animation

The main work is done, but we still need some improvements. First let’s add a second variable to define the canvas width. It makes more sense to use a fixed value here. So we can render even long text on the small canvas.

We just call it “width”. And we also want to start the ticker on the most-right position of the canvas. That’s why we take the width of the canvas we just declared and use it as a substractor.

 1=LET(
 2    ticker; "hello world"; 
 3    width; 10;
 4    length; LEN(ticker);
 5    inc; IF(
 6        INDIRECT("Z" & ROW() & "S" & COL(); 0)
 7            > (length + width);
 8        1;
 9        INDIRECT("Z" & ROW() & "S" & COL(); 0)
10    );
11    MAP(
12        SEQUENCE(1; width);
13        LAMBDA(val;
14        IF(val = 1; 
15            val +  inc;
16            IF((val + inc) > (width - 1); 
17                MID(" " & ticker; val + inc - (width - 1); 1); 
18                "")
19        )
20    )
21)

Please note how the width-variable replaces length as an argument for SEQUENCE. We also need to adapt the inc-part a little. We need to reset the incrementation a little later, because the animation should start at the most right column and therefore needs more steps to finish.

The second and nested IF-condition checks if the current incremented value of the particular cell is greater than the width of the canvas. That ensures that we address a valid value from the ticker text when using MID.

And that’s it. I adjusted the cell widths a little, hid grid lines and made the first cell invisible to make it look a little better. This is your ticker:

You can just copy and paste the formula to every sheet you want or you can create a LAMBDA function to get a handy formula. Do not forget to enable “iterative calculations” and change the ticker text.

Have fun.

Nota bene

This does not work in Google Sheets, out of the box, because Google Sheets do not support the LET formula. You still could just fixed values or named ranges. But this makes it less fun.

I work extensively with AI, and as it is a new tool that introduces an entirely different way of working, it sometimes takes time to adapt to these new circumstances. In this context, I am still exploring the best approach to integrate AI or agentic chatbots into my development workflow. Recently, I reached another milestone that I’d like to share.

Preliminary notes: You may think that “the plAIbook helps you to vibe code everything. You are wrong. You still need a profound understanding of architecture and design patterns. The more complex a project gets, the harder it is to maintain with AI only. This is not a fire-and-forget solution.

Also, as this is not the only but main focus of my developing work: This is slightly focused on creating little PoC`s using Python and/or Flask. But I believe it’s quite easy to adapt the process for other languages, frameworks or platforms.

What are the landmarks here?

• a meta prompt that returns your necessary prompts for your actual coding Agent
• a fully fledged pre-prompt instruction
• an environment that makes it easy to refactor or dive into the code on your own
• an MCP server (yeah, buzzwords are required here) that helps the Agent to supervise its own work

You will find a more technical summary, all prompts and scripts there: click me

This said, with no further ado, let’s dive into the details.

The Process

The developing process is separated into three phases:

Phase 1 - Prepping

First we need to prep our environment, means: Setup Visual Studio Code. I assume that you have a Copilot subscription. Though, the whole process will work without it and it should also work with one of the derivatives, like Windsurf or Cursor.

1. Add pre-prompt instructions

Assumining you have created an empty folder, declared it as a workspace and inited a git repo there, this is how you proceed:

Take the pre-prompt instruction ( from here ) and put them to .github/instructions/global.instructions.md.

Hint #1: If you click the cog wheel at the top of Copilot’s chat window and select instructions you can add even more specific instructions. They will be placed in the very same folder.

Hint #2: As these are global instructions, you may also want to place them to copilot-instructions.md - which is the native VSCode way - I prefer the first way, as this allows you to maintain multiple instruction sets.

As you can see the pre prompt uses a lot of abbreviations and doesn’t seem very natural. That’s because it’s sufficient enough for the machine to understand it’s task. However, let me guide you through some benefits and features I built in:

• functions are not bigger than 20 lines, files don’t exceed 200 lines
• I am pursuing a modular approach with features separated into files and subfolders
• code must be human readable with appropriate inline documentation, files are structured and grouped into functional sections
• common security, logging and error handling measures are implemented, like of course try/except blocks, rate limiting, sanitizing inputs and so on
• the agent creates commit messages on its own after every minor change
• extensive tests for functions and API endpoints
• lean environment with minimal dependencies
• extensive but not bloated documentation, with a PROCESS_MAP.md showing the overall architecture and flow of the application or an UNFINISHED.md file outlining current work, which is quite helpful if the agent stopped working or you want to get back to the current state at a later point
• Docker implementation which easily enables you to take everything and run it anywhere else
• utilizing MCP to maintain supervising tasks, cleaning up, testing and so on
• telling the agent to be sceptical and ask user for additional details
• trying to prevent “bug looping” where the agent implements the same fix multiple times without success (this is another reason why UNFINISHED.md exists)
• and last but not least but most important: No useless “Sorry” & “Apologies” or offering “This is the final solution” - you know how annoying this is, when you read it the twelfth time and the app is still not running ;-)

Note: As already said: This has a strong focus on Python and Web app development, but most parts are also easily applicable if you want to, like, create a pure JavaScript app or even mobile apps.

2. Enable MCP

The model control protocol (MCP) is a fancy name for an interface that the Agent can talk to. On one hand it may help you to control other programs, like imagine connecting the Agent via MCP to Adobe Premiere and telling it to cut your home videos. We are using it to allow the Agent to supervise and revise its own output!

First create a new file .vscode/mcp.json with this content - I will explain it in a second:

1{
2  "servers": {
3    "ai_coding_agent": {
4      "type": "stdio",
5      "command": "python3",
6      "args": ["./.mcp_tools/mcp_server.py"]
7    }
8  }
9}

Now create a folder named .mcp_tools and put the following files into it:

• mcp_server.py - the server itself
• load_context_files.py - allows the agent to refresh its pre-prompt brain
• run_tests.py - allows the agent to run tests against the app
• update_process_map.py - allows the agent to update the process map

(You may restart your Visual Studio Code)

Now open the command palette to find our MCP server (MCP: List Server) and start it (or use the very handy inline-commmands VSCode is offering you). The output window should return something like that:

Hint: In some environments the command may need to be adjusted to python instead of python3, depending on your Python installation.

Perfect, the server is running. You can conform that by clicking the little cog wheel at the top of your chat window, which opens the extension bar (usually on the left) - and as you can see, VSCode already offers a lot of different MCP solutions. At the very bottom is our own:

Test your agent directly using this tool reference, which acts as a magic word, to reload pre prompt:

1#load_context_files

which should be confirmed properly by the bot (you can use natural language too, but I prefer calling tools directly).

I suggest you checkout the mcp_server.py which is not too complex. It defines tools the MCP server offers to your bot, right now there are three: load_context_files, run_tests and update_process_map

Hint: You may have noticed “Toolset” - You can combine sets of different MCP tools into one toolset to perform multiple actions. Also, if you look for more sophisticated interactions, of course you can feed information into your MCP server tools. Like not only telling it to run tests but actually provide data to run the tests with and submitting them as arguments.

Phase 3 - Planning

In this phase we will create 3 to 5 sequential, self-contained prompts. I mean, not we, but the AI, apparently. For this I created a “meta-prompt”. Throw it against your most favorite AI agent and wait for the result. This is a strategy that I suggest to everyone. Don’t try to wrap your head around writing the perfect prompt. Ask someone who should know it - better than you: the LLM!

Hint: There’s one cruccial information that you should take with you at this point: Don’t was time working on prompts on your own, manually. Utilize the power of AI to get the perfect prompt!

So, we can keep this short: use the meta prompt to get a small handful of prompts for your actual developing progress. Those prompts are reflecting different phases of your developing process.

You’ll find my meta prompt here . Just fill out the four required fields:

• [NAME] - of your application
• [GOAL] - what’s this app about?
• [FEATURES] - what does it offer?
• [USERS] - who are the target users?

Keep goal and features tight. Don’t fall into the feature creep rabbit hole. Your goal should be a good working first draft of your app. Focus on that and then add features one at a time.

Now run your meta prompt and collect your actual prompts for your Copilot agent.

Phase 4 - Execution

The rest should be a walk in the park. (Until it isn’t (˵ ͡° ͜ʖ ͡°˵)). Apparently you start pasting your prompts to the agent and watch it working the task.

You may leave it working like that, but I strongly suggest to keep watching. When it’s done, don’t just paste the next prompt. Try to understand what has been built here, not in detail, but for later troubleshooting and feature expanding, it’s quite helpful to understand the architecture. You can always ask the machine, but it will probably not know the great picture.

As mentioned above, the pre prompt covers a lot not only technical issues but also tactical directives that should improve how the agent works and talks to you.

A miner has to solve a mathematical problem to validate a block for the Blockhain. The so called Proof of Work requires a lot of computing power aka energy and therefore is quite expensive. Googles BigQuery has a price model, that charges you for the transferred data and not the computing power. Meaning: You can utilize a lot of computing power for free. If you are able to reproduce the mining algorithm in SQL.

In this article I first will explain how the mining process works. This is the main concept for a couple of Blockchain technologies. My explanation is based on Bitcoin’s blockchain. If you are already familiar with that, you may skip to part 2, where I present an SQL query that does the job.

Part 1: The mathematical problem aka: Mining a block

What’s a hash again?

A hash is a kind of unique piece of information calculated from any given input ( I wrote about that earlier ). There are a lot of hashing algorithms. The blockchain we are talking about uses SHA256. It’s important to understand, that a hash is basically just a really large number with always the same length: 256 bit.

If you change the input, even just a little, the hash changes. This makes the hash a pretty good indicator to check if the input data has been tampered: If you know the original hash and you re-calculate the hash for a given information, and both hashes are different, the initial information has been changed.

Why does the blockchain rely on hashes?

Every block in a blockchain contains two kind of information: A couple of transactions details and a block header. Among other details the header contains two important hashes: One hash for all the transactions of the current block and one hash for the header of the previous block.

If you would tamper the transaction data of lets say block #1337, the hash of this block’s header would change. But as the next block’s header contains a hash for the original block #1337, you could simply compare both hashes and come to the conclusion, that the data in block #1337 has changed.

This sounds easy, because calculating one hash only takes milliseconds, even on slow computers. That’s why the blockchain protocol defines a very special requirement that every header’s hash has to meet: The hash of a block header has to be lower than a given number, the target.

S ide note: Besides that there are some other rules that make the calculation even more difficult: For example you need to double-hash the information and you need to calculate the Merkle root of all transactions of the current block.

You can meet this requirement by simply adding an incrementing number to the header information, re-calculate the hash, and see if the new hash is below the target.

The whole process is so complicated, that even powerful computers will take around 10 minutes to find a solution. That’s by the way a reason why Bitcoin is not the payment method: If you’d like to buy a coffee, you may have to wait around 10 minutes until your transaction is confirmed.

The block header

Let’s see what kind of information a block header contains. This is the header of block number 727938 :

You will find six important fields here:

• a version field,
• the hash for the previous block’s header,
• the Merkle root,
• a timestamp,
• a bits field, that defines the target and
• a nonce

Let me describe those fields in detail:

The nonce field

If you take all header fields except the nonce field, you get the block message:

104c04d34702a058fee2c141a685dfc3d18c4f0efc0464d625057060000000000000000005e578fe5fe001059710edbf9f78c1bd15b110faf574723cafde35667633c18084ac23462c0400a17

The SHA256 hash for this looks like that:

15c96f247d3c6a48b244be859d888dd1ee2a80dd6378cc7e10f31b9d14b78660b

As mentioned above, the protocol requires you to add a specific kind of information to the block message, that leads to a hash below a given value. For starters lets make that easy: Find a number that leads to a hash starting with a zero. What if we add a 1 to the block message? (Remember: This is a hexadecimal number, so I need to add two digits):

104c04d34702a058fee2c141a685dfc3d18c4f0efc0464d625057060000000000000000005e578fe5fe001059710edbf9f78c1bd15b110faf574723cafde35667633c18084ac23462c0400a17**01**

That’s the resulting hash:

1855ccd41ef49272ab633eff0cf5abed6d543ed5183eaf9f8a32e12a57b7419a9

No leading zeros? You may get a picture of how difficult it is. The actual target value for this block was a hash with 19 leading zeros. The miner need to (find and) add the number 1.396.904.612 (in hexadecimal 0x534312a4) to the block message to find a fitting hash. And that number is called **nonce **— short for number used once. That’s the resulting hash:

100000000000000000006b9bf976b004581c19d800948d2eab77c8f9e07b9f088

While calculation one hash seems pretty simple, imagine how long it would take for 1 Billion calculations, or even more? Even more? Yes, you heard right: The nonce is a four byte number meaning you have 2³² possible numbers (4,294,967,296) to check (the so called search space). And sometimes that’s not enough.

The previous block’s header hash

This is just the hash of the header of the previous block. Not much to say here: You cannot change it.

The bit field

This field defines the above mentioned target. It’s a 4 byte number, too:

10x170a40c0

A hash is a number with 32 bytes. And that’s strange: How can you limit a 32 byte number using only 4 bytes? Here it is: The first byte of the bit field defines an exponent: 0x17. The remaining 3 bytes define the mantissa: 0x0a40c0. The base is 256 and you need to substract 3 (length of the mantissa) from the exponent. So this is the formula to calculate the target:

1target = 0x0a40c0 × 256⁽⁰ˣ¹⁷ ⁻ ³⁾⁾

The target will be adjusted every 2,016 blocks, which is roughly two weeks (6 blocks per hour × 24 hours × 7 days x 2 = 2,016). Why? The mathematical problem needs to be difficult enough to keep the chain safe but it must not be to difficult, because you do not want to wait more than 10 minutes for a transaction to be confirmed. As computing power changes or miners joining and leaving the network, the target and therefore the difficulty as to be adjusted.

The Merkle root

The Merkle root is a hash of all transactions of **the current block **(and a little more, I’m not going to much into detail here). Basically it’s pretty easy: You take two transactions and hash them to get Hash #1. Then you take two more transactions, hash them and get Hash #2. Now you take Hash #1 and Hash #2, hash it and get: Hash #3. You’re building an upside tree, where the transactions are on the bottom and the branches are dozens of hashes eventually leading to the root hash.

If the search space is not enough, the miner can change the order of the transactions to create a different Merkle root. This would change the block message and therefore lead to a different hash.

The timestamp

The timestamp, well, shows at what time the block was validated. In fact there is a lot more to tell about the timestamp, but not in a mathematical way.

The protocol uses the timestamp to see, how fast blocks are mined. As already explained, the network uses this value to adjust the difficulty.

A miner can use the timestamp field to increase the search space. A valid timestamp has to be greater than the median timestamp of the last 11 blocks and lower than the network-time plus 2 hours. Eventually this gives you a 3 hour window where you can adjust the timestamp to gain extra search space. 3 hours are 10,800 seconds (which is roughly 2¹³).

The version field

Last but definitely not least: The version field contains a lot more than you would expect. The version field is 4 bytes long and it

• contains the current protocol version,
• signals readiness for soft fork proposals and
• offers additional search space.

That’s a lot of information and it’s not that easy to pull it out of those 4 bytes. The current protocol version requires the **first 3 bits **(!) to be at least 0b001 (2 in decimal):

Besides that, the last 13 bits are used for soft forks signaling. This simply means, if a miner is ready for a particular soft fork, he flips on of those bits to 1. Imagine there’s a soft fork BIPx that requires the 1 bit to be flipped, that’s how the miner signals readiness for BIPx:

This would lead to a version field like this:

10x20000001

As you may have noticed, this leaves 16 bits open (32 bits at all, mins 3 bits for the version and 13 bits for soft forks = 16 bits left). This gives the miner 2¹⁶ more options to find a good hash! That technique is called overt ASIC boost. Lets take the version field from block #727938 for example: 0x344dc004

This is the bit representation of the hexadecimal value:

So apparently the miner took 0b1010001001101110 (0d41582) as an additional resource to finally find the fitting nonce 0xa4124353.

Block header summary

So after all you have six fields in your header, where only two fields are fixed and cannot be changed in order to find a hash below the given target: The bit field, that describes the target itself and the hash for the **previous header — **the vital element of the Blockchain concept…

Besides that you may change the nonce, the one field actually designed to change. The Merkle root, by re-arranging or considering different transactions. The version field and the timestamp by exploiting the 3 hour time window.

To give you an idea how complex it is to utilize at least the nonce and the version field, I created a Python script that does that for you:

Part 2: Using SQL and BigQuery to validate a block

Now as you are an expert in mining, let’s try to find a way to mine for free. I will implement a mining algorithm in SQL that iterates through the nonce and the additional version bits to find a hash below a required target. I’m not re-arranging the Merkle root and I will take the timestamp as a given value.

To make the whole process understandable, I will start from within the inner loop and calculate the hash from a given header, that already contains all the correct “answers”. I’m using the header for block #727155 with all the information already in the right format. After we understand the hashing process in BigQuery, we will build the header field by field and finally add the two loops.

And here they are — the variables for my six header fields. Remember that the version field contains the actual version as well as the soft fork signals.

1# header for block 727155DECLARE hash_prev_block STRING DEFAULT "00000000000000000004136135b2e0cd367b56ea6c0dd5b8f79964a4cd7d2718";DECLARE merkle_root STRING DEFAULT "0d14fac91555d6337b10b2f20de231858fb5225f2ff685cd9b487c235d6e8307";DECLARE header_datetime STRING DEFAULT "622defb9";DECLARE bits STRING DEFAULT "170a3773";DECLARE nonce STRING DEFAULT "c8f05860";DECLARE version STRING DEFAULT "20002000";

We simply need to concatenate everything into one string before we hash it. Simply? Wait…there’s one pitfall called Little Endian. Bitcoin wants you to work with values in Little Endian. This is the way how you can write a number. In one sentence: Little Endian starts with the lowest value on the left. Basically you just switch the order of the single digits in your number. Instead of 23 you write 32. The value though remains the same. You achieve that using the REVERSE() function in SQL.

But beware — there’s more: What would you expect if you reverse the following hexadecimal value to Little Endian?

1ABCDEF

Is it FEDCBA? No. We are working with bytes. One byte is 8 bits which is enough space for 255 values or in hexadecimal: FF. Two digits. That means we have to see all them values in pairs of two and the correct Little Endian representation is:

1EFCDAB

To get this result you need to read the incoming value in bytes format and afterwards transfer it back to hexadecimal string. So this is how we create the actual header:

1DECLARE header STRING;SET header =    TO_HEX(REVERSE(FROM_HEX(version))) ||    TO_HEX(REVERSE(FROM_HEX(hash_prev_block))) ||    TO_HEX(REVERSE(FROM_HEX(merkle_root))) ||    TO_HEX(REVERSE(FROM_HEX(header_datetime))) ||    TO_HEX(REVERSE(FROM_HEX(bits))) ||    TO_HEX(REVERSE(FROM_HEX(nonce)));

And that’s the header containing all relevant fields in Little Endian:

10020002018277dcda46499f7b8d50d6cea567b36cde0b23561130400000000000000000007836e5d237c489bcd85f62f5f22b58f8531e20df2b2107b33d65515c9fa140db9ef2d6273370a176058f0c8

The Hashing

BigQuerys build-it SHA256 expects a string or a byte but the header is made of hexadecimal values. So first we need to prepare the header, meaning: convert the hexadecimal string to the bytes format:

1DECLARE b_header BYTES;SET b_header = FROM_HEX(header);

Now we can apply SHA256. And as mentioned above, we need to do it two times. The result of SHA256 comes in bytes, o we have to convert it back to a hexadecimal string:

1SELECT TO_HEX(SHA256(SHA256(b_header)))

That finally leads to this result:

1f0d400dd8daaffeaa31a369c41bb6b7982b15ad47a8c07000000000000000000

And now you hopefully remember that we were running everything in Little Endian. To finalize the process we need to convert it back to Big Endian and finally get to this hexadecimal number:

1SELECT TO_HEX(REVERSE(SHA256(SHA256(b_header))));

And that’s the result:

1000000000000000000078c7ad45ab182796bbb419c361aa3eaffaa8ddd00d4f0

that you now may compare to the actual header hash the next block refers to :

Prepare the inner loop

We’re going to wrap everything into a while-loop, add a break-out condition for safety reasons and start with a comparison of the just found hash and a target. Because that’s what it’s all about: Find a “number” (hash) that is lower than another “number” (target).

The target value is hidden in the bits field. And as already mentioned, we need to do the math to get the numeric target. The mantissa comes from the last 3 bytes, the exponent is the first byte. And the target comes from this equation:

1target = mantissa * 256 ^ exponent

And this, well, leads to a pretty big number. Actually to big to fit into an INT64. We have to store it into a FLOAT64 and to make it comparable against the previously calculated hash, we need to cast the hash to FLOAT64, too.

Here’s the loop containing the condition, that returns the nonce — that we are looking for and a little break-out-statement, because we’re just testing how it’s working:

1# skipping previous declarations from above# [...]DECLARE mantissa INT64DECLARE exponent INT64DECLARE target FLOAT64;DECLARE max_iterations INT64 DEFAULT 100;DECLARE iteration INT64 DEFAULT 0;SET exponent = CAST(CONCAT("0x", SUBSTRING(bits, 0, 2)) AS INT64)SET mantissa = CAST(CONCAT("0x", SUBSTRING(bits, 2, 4)) AS INT64);SET target = mantissa * POWER(256, exponent);WHILE true DO    SET header         TO_HEX(REVERSE(FROM_HEX(version))) ||        TO_HEX(REVERSE(FROM_HEX(hash_prev_block))) ||        TO_HEX(REVERSE(FROM_HEX(merkle_root))) ||        TO_HEX(REVERSE(FROM_HEX(header_datetime))) ||        TO_HEX(REVERSE(FROM_HEX(bits))) ||        TO_HEX(REVERSE(FROM_HEX(nonce)));    SET b_header = FROM_HEX(header);    SET header_hash = TO_HEX(REVERSE(SHA256(SHA256(b_header))));    IF CAST(CONCAT("0x", header_hash) AS FLOAT64) < target THEN        SELECT nonce;        BREAK;    END IF;    IF iteration > max_iterations THEN        SELECT "No nonce found";        BREAK;    END IF;    SET iteration = iteration + 1;END WHILE;

Complete the loop

Yeah well — isn’t that something? The only thing we need to do now is to actually iterate through the nonce to let the query find the correct value. To have an iterable variable we need one of the type FLOAT64. I will set the value a little bit under the expected target, to speed up the testing:

1DECLARE nonce_dec FLOAT64 DEFAULT 3371190464; 

Now, after the loop starts and before we collect all header details,we need to get the hexadecimal nonce. And at the end of the loop, we increment the decimal nonce:

1WHILE true DO    SET nonce = FORMAT("%X", nonce_dec);    SET header =        [....]    SET nonce_dec = nonce_dec + 1;END WHILE;

Adding the second loop

Now as we already learned, it may not be enough to test 2⁶⁴ possible numbers to find a fitting nonce. That’s why we’re going to add another loop to use the version field for some additional search space. Unfortunately this is a little more complicated. So, what do we need? Binary numbers!

This is the version for our famous block # 727155. Again we’re using this as an example. If you’d like to mine a new block, you need to adjust at least the soft fork part.

10x20002000

In binary that is:

100110100010011011100000000000100

The last 13 bits are representing the soft work readiness. That’s a given value. The version is pre-defined to be (at least) 001 for the first 3 bits. We’re taking the easy route here and are going to declare this in binary already. The remaining 16 bits are up and ready to be iterated through:

1DECLARE soft_forks STRING DEFAULT "0000000000000";DECLARE version_roller INT64 DEFAULT 1;DECLARE version_roller_bin STRING;DECLARE version_bits STRING DEFAULT "001";

Please note, that I’m setting the version_roller to 1— that’s the one used in the block we are working on. Again, this is to speed up the demonstration here.

We will use an additional outer loop to iterate over the version roller. The iteration comes at the end, at the beginning of the loop we need to process the decimal value to get 16 bits out from it and press it into the hexadecimal format:

1main_loop: WHILE true DO        SET version_roller_bin = SUBSTR(bqutil.fn.to_binary(version_roller), -16);    SET version = FORMAT("%X", bqutil.fn.from_binary(CONCAT(version_bits, version_roller_bin, soft_forks)));WHILE true DO        [....]    END WHILE;    SET version_roller = version_roller + 1;END WHILE;

Take care of the label I set here to address the loop: MAIN_LOOP. I have to reference the outer loop when breaking from inside the inner loop:

1    [...]    BREAK main_loop;    [...]

Check out the full script in final part.

Congratulations and Conclusion

Basically that’s it. That’s how the full script should look like this ( gist.github.com ). Feel free to increase the value for max_iterations and set the starting value for nonce_dec to 0 and see how long it will take Google to find the correct nonce.

This is a lot of weird SQL stuff and far away from SELECT * FROM table. Is it worth it? Well, I set the starting nonce to 3370194460 and executed the script.

Spoiler Alert: After 10 minutes I stopped the query. Another approach (see below) seems way faster, though it’s a cheating with the actual calculations.

Disclaimer

I will not loose much words about Proof of Work. The process itself is sophisticated and somehow thoughtfull… as well as a little weird. But above all it’s also a huge wast of energy and there are better ways to achieve the same outcome. I’d like to end with a quote from another author on Medium who said:

You’ve got to wonder what we’d achieve if we used even a fraction of this computing power for medical research instead…

Additional ressources

This is a pretty good and helpful medium article about the Blockchain header by RJ Rybarczyk. This is a website that provides a lot of technical background information as well as tools to calculate hashes or browse the Blockchain.

A couple of years ago Uri Shaked also [wrote about this topic](I stumbled over his article while doing research. But he was pursuing a different approach. First of all he was using a random feature to find a fitting nonce. I also considered this before, but as you cannot ensure to ge the same random value twice, you will waste a lot of time (he actually calculated the same nonce within 20 seconds, which still is faster than my approach). Besides that he was using sub-queries to create temporary tables filled with the nonce.

I’m not a fan of mobile games

Not because I don’t like playing, but because a questionable business model has developed around mobile games, making it very difficult to find the true gems. It seems like it’s more about selling ad space or generating in-app purchases with simple game mechanics. Of course, there are exceptions, but they get lost in the sheer volume of offerings.

I recently stumbled upon one of these games by chance, and it perfectly represents these accusations: Dozer Demolish .

On Google Play, the game is advertised without an age rating, while on Apple’s App Store, it’s rated 4 years and up. The Android store mentions over 10 million downloads, according to developer Eternal Studio, it’s more than 15 million .

The game is distributed by HOMA , which also contracts other developers. Headquarters: France. So within the jurisdiction of the GDPR. The motto of HOMA is:

“Amazing Content” - Well, let’s see!

Games in the “Convenience Gaming” category follow the same principle: There’s hardly any depth, the rules are quickly explained, the controls are intuitive, and often not much skill or thought is required. Everything is geared towards short-term dopamine release. This doesn’t have to be derogatory; it’s a form of entertainment that attracts millions of players.

The gameplay of Dozer Demolish adheres to the industry’s principle. You roam with heavy machinery across changing landscapes, destroy buildings, collect their remains, and earn money to buy new vehicles or upgrade existing ones. One-handed and on the side. On the toilet. In bed. On the train. During meetings. “Convenience Gaming,” indeed.

The thing about data privacy

So far, so unspectacular. Things get interesting, provided you have a penchant for such details, when it comes to data privacy.

According to its self-disclosure, the app shares data in the following categories with other companies:

• Financial data
• Personal data such as email addresses and user IDs
• App activities

Not inherently reprehensible, as this makes sense for in-app purchases and advertising.

But location data? (Hint: Location-based advertising…)

Problematic. But that’s not all. Under security measures, it states:

Data is not encrypted. Data cannot be deleted.

Okay. At least honest, if not up-to-date. On Google’s Play Store, the developer’s privacy policy is referenced, which hardly deserves the name “privacy policy.” A Google email address, a paragraph on Google Analytics, and then a note on “minors”:

“Eternal Studio does not knowingly collect personal information from children under the age of 13.”

Of course not. Too bad the game is offered without an age rating.

“If Eternal Studio learns that Eternal Studio has inadvertently gathered personal information from children under 13 years of age, Eternal Studio will take reasonable measures to promptly delete such personal data from Eternal Studio’s records.”

“inadvertently” - that sounds a bit like shirking responsibility.

This is only surpassed by the following note:

“If you are under the age of 13 or a minor in your country of residence, please ask your legal guardian’s permission to access and use our services.”

So the 4-year-old child is supposed to read the English privacy policy and then ask their parents for permission. Hilarious. Reminder: The game has an enormous reach with 15 million downloads. A bit more professionalism isn’t too much to ask, is it?

Needless to say, the site provides no information about who you’re actually dealing with. According to the “Impressum,” you can contact them via xxx@eternalgamestudio.com . In the privacy policy itself, it’s xxx@gmail.com . [The Facebook page](refers to xxx@gmail.com . [On X/Twitter](you at least learn that the developer studio is presumably based in Michigan (USA).

(Email addresses anonymized, although they are publicly accessible, but who knows…)

Enough complaining; perhaps — and I say this quite sincerely — the developer was surprised by the success, and I just happened to stumble upon a rare exception. Nevertheless, the game is backed by a fairly large publisher. A product with 15 million users and such an aggressive advertising strategy deserves closer scrutiny.

(The Apple App Store, by the way, refers to [the publisher’s privacy policy](It’s casually written and quite detailed.)

At first glance, formal data privacy already fails. What about the app itself?

It starts with the usual consent dialog. Here, you can either accept everything unseen or influence the collected data via detailed selection. Problematic: There’s no button to “reject all.” The detailed selection is hidden in a second view and is not immediately accessible.

The list of technologies used is impressively long for a game. 69 vendors measure ad performance, 24 measure app performance. 33 providers analyze target groups, 57 improve services. 7 providers handle targeting.

Each category is optimistically declared as “legitimate interest.” Reminder: The game is offered without an age rating. For minors, special protections apply under the GDPR.

Even the individual processing categories must be manually deselected. Needless to say, the explanations are anything but understandable for a normal user, let alone a minor. (Incidentally, the industry association IAB has done an excellent job here, selling this kind of overspecification called TCF (Transparency and Consent Framework) as “transparency.”)

Legitimate interest? Please manually deselect!

Transparency doesn't mean comprehensibility

But that’s not all: In the app, you can’t correct your selection afterward; there’s simply no menu option for it. This is also hardly compatible with GDPR requirements.

The app doesn’t leave the best impression regarding data privacy. But that’s not all, let alone what makes a game.

(And here we are again with my general criticism of data privacy: Most users are probably more annoyed by the endless consent banners than feeling informed. Complete data collection avoidance isn’t feasible either; as a data analyst, I’m aware of this. But smaller studios and companies are likely overwhelmed by the legal requirements.)

Never mind. What about the content?

The thing about advertising

As mentioned above, a business model has established itself here where free games seemingly only serve as platforms to push advertising onto devices with — attention, subjective exaggeration for dramatic effect — a blatant lack of subtlety. It reminds me of a not-so-famous quote that I once fittingly recited :

Journalistic content is the vehicle to capture the audience’s attention for advertising content.

Springer lawyers in 2015

And here, it seems no different. From a business perspective, I respect the success. From a user’s perspective, this practice is only a hair away from being a scam.

The game is offered for free. For €1.99, you can purchase an ad-free package. This is at least misleading — more on that shortly — and technically poorly implemented:

I bought the package on one device, but on a second device, ad banners still appear at the bottom of the game.

The entire advertising isn’t even disabled. You get three dozen “upgrade tickets,” which finance certain in-game actions. After about 1 hour of gameplay, these are already used up. Then, upgrades and other game mechanics sporadically but also with intrusive regularity require ad videos.

And these are intense. We’re not talking about small 30-second inserts like on YouTube.

“Dozer Demolish” usually plays two spots, totaling almost 2 minutes. Some of the advertised games can even be played directly for about 15 seconds. Can you close the ad overlay now? No. First, you’re taken to the Play Store. From there, you return to the ad overlay, with a final ad message that can now be closed.

Wow.

The first insert usually lasts 60 seconds.

Then another one, about the same length.

Since basic game mechanics are affected by these inserts, you easily spend 2 minutes playing and 2 minutes under ad pressure. If you don’t want this, you can replenish the upgrade ticket quota. For example, 10 tickets cost €6.99. This corresponds to about 10 minutes of gameplay under normal conditions. The most expensive package costs €9.99 and contains 60 tickets.

In-app purchases in "Dozer Demolish" (Source: iOS App Store)

The best value: 60 tickets for €9.99—2 minutes of your life cost you €0.16.

Special offer! €6.99 for 10 tickets and some upgrades. The price of your life increases to €0.69!

What’s the problem?

“Well, everyone can decide for themselves!” one might say. What’s the problem? As mentioned, it’s a brilliant business model. The problem is primarily the age rating and the handling of data privacy requirements. Secondly, from a personal perspective, the excessive coupling of game mechanics to ad delivery.

Do you want to develop games or deliver ads?

Problem 1: Legal capacity of minors

With an age rating of 0 (or 4 years on iOS), this game is accessible to a group of people who, at least under German law, are not legally capable. Only at 7 years old are they partially legally capable and can make small purchases (“pocket money clause”) without parental consent. How does this align with the sometimes very expensive in-app purchases?

Google describes the age rating of 0 in two different ways:

or way less informative:

Problem 2: Dark patterns

The game uses dark patterns to place ads, such as when transitioning to a new day. A large green button starts an ad block and rewards with in-game currency. The small text below skips it. Many upgrades are advertised as “free,” but they usually involve an ad insert. Does only my moral compass sound the alarm when this practice is applied to minors?

Please press the big green button; it won't be to your disadvantage!

Behind this button is an ad insert. Would you have recognized it?

Problem 3: Accessibility of games with higher age ratings

The ads are not only intrusive; they are often far from age-appropriate! This is especially problematic when a minor has the opportunity (not to say “is forced”) to test a game for 15 seconds, and the advertised game has a different age rating! Imagine going to the cinema with your child to watch Cars 7, and before the movie, there’s a trailer for Texas Chainsaw Massacre.

Hey kid, still unsure? Test our game now. Or resist the temptation for 15 seconds.	The game is rated 12 years and up. But you don't have to tell your parents.	This game can also be tested during the ad break.
Rated 12 years and up here as well.	Every 4-year-old loves it. Shooting.	And so it's not just watching: Shooting yourself makes you want more!
Rated 12 years and up here as well.	A Monopoly clone is also advertised. Rating here: 16 years	Can't be missing as ad entertainment: A zombie game, rated 16+. Sweet dreams tonight, little prince.

Guess why Monopoly is rated 16 years and up? Not because of violence, apparently, but because of incentives to buy and in-game purchases.

Sad isolated case?

I stumbled upon Dozer Demolish by chance. Homa currently offers 50 games in the Google Play Store. I looked at four other games.

All in Hole! . Here, ad delivery is much less aggressive. After 15 minutes of gameplay, there were no ads, not even as an overlay at the bottom of the screen. But the in-game purchases are hefty. The largest package costs €99. On consoles & PC, you get high-quality triple-A titles for that. Interesting: The age rating is 12 years. That doesn’t really make sense, as I didn’t see any ads here and the gameplay is anything but “not suitable for minors.”

[Aquarium Land]((from 0 years, 50 million downloads). Here, the first ad appears after 5 minutes. Again, a game from another age category is advertised.

[Cube Blast Journey]((0 years, 1 million): This game comes without a consent dialog. That’s unusual. And indeed: monitoring diligently records requests, some of which clearly count as tracking and analytics requests. For example, those to gameanalytics.com, amazon-adsystem.com, applovin.com, adjust.com, or doubleclick.net.

[Solitaire Klondike Classic]((0 years, >5 million). The game is distributed by Homa, but after a short time, you’re referred in broken German to “[the new version]((0 years, >1 million),” distributed by Freelax . Also, due to the banner at the top of the screen, it’s not really clear whether this is a legitimate reference or simply advertising.

In Solitaire Classic, right after the familiar TCF consent dialog, there’s another consent request — this time preselected. Since GDPR, this is no longer permitted, as it doesn’t constitute “explicit consent.”

The supposed successor is even bolder regarding consent. Here, starting the game is tied to consent—which is also legally questionable.

[Cat & Mouse]((6 years, 10 million). Here, too, ads are actively displayed, referring to apps and games that are not age-appropriate.

What about other publishers?

Remarkably, the games advertised that I mentioned previously do not pursue a comparable advertising strategy, in fact some do not even offer in-game ads at all. Which doesn’t mean that the publisher behind them doesn’t have assets in stock following the same principle. Let’s look at [Century Games PTE. LTD.](the publisher behind Kingshot. They have 23 games in their library, of which 12 have in-game ads (according to their Google Play Store listing), 5 with no age restriction. I checked these three:

[Tasty Travels: Merge Game](comes without a valid consent dialog, but according to their data privacy policy collects data for several purposes, including marketing. Eventually, after playing for a while, I was presented a playable demo for [Whiteout Survival](([Century Game PTE LTD](age rating 12+).

Playable ad for Family Farm Adventure

Playable ad for Idle Courier

Family Farm Adventure has a questionable consent dialog, like the one that Dozer Demolish offers. While I didn’t notice any ads at all, the prices are even worse than experienced before. 120 Euros for an in-game package - without knowing how long this lasts, this is clearly nothing you want to offer to minors.

[Idle Courier](also comes with this not-so-really-transparent consent dialog and the “coupled consent” (if you go on, you accept our privacy policy). In this game it only took me 2 minutes to face the first playable ad for Kingshot (rating 12 years and up).

River Game and Funfly PTE do not offer any games with an age rating of 0 years. FunPlus International AG has 15 games in stock. All of them with age ratings between 12 and 16 years. Scopely has just one game with age rating of 0, but without ads.

Conclusion?

The games mentioned pursue their advertising goals far less aggressively than “Dozer Demolish”, but the delivery also seems to suffer from the same problem: Although a certain age restriction is given, ads for games for older ages are displayed. And not only that, they can even be tried out directly.

Now, one can argue whether 4-year-old children should even have access to tablets and smartphones to play such games. But that’s a pedagogical question. Ultimately, the games are distributed with a certain age restriction, but still pave the way to content that is not age-appropriate. That’s no longer just a pedagogical issue. And I am not even talking about consent dialogs or price ranges way beyond reasonability.

It’s remarkable that such successful games fly under the radar. From my own experience, I know that Google at least formally has very strict requirements. Compliance with platform guidelines is (automated?) monitored and enforced with deadlines and the threat of de-publication. That’s my impression. Is Google sometimes a bit more lenient because its own revenue depends on advertising income?

Don’t get me wrong: Most of the games have quite a high quality, some even with a decent story behind them. It’s only fair that the developer or publisher tries to monetize the game to cover development costs. Also, I only stumbled upon Homa by chance, and I don’t want to portray the publisher as a black sheep here. But as this is about the youngest target group, lax data privacy practices, ads beyond control and expensive in-game purchases - I think it’s worth taking a closer look.

Some games make me wonder: Is it still about delivering good entertainment or just flooding the market with low effort to keep users glued to the device with simple game ideas for as long as possible, in order to show as much advertising as possible? Advertising that is no longer just the blinking banner at the edge of the screen, but interactive content that demands more time than the actual game?

I’m not a fan of mobile games. And now you know why.

Statement from Homa and Google

On 30.07. I asked Homa and Google for a statement. Homa responded within 2 days and announced that they would address the issue internally. Exemplary, I’m curious. Google has not responded so far.

Chatbots are going to replace search engines. Aren’t they?

Well, no. I asked Gemini (Google’s AI chatbot) how to update Stadia’s firmware (Google’s former gaming platform). Gemini suggested searching the internet for a proper solution:

Don’t they talk to each other?

You shall not pass

We all struggle with captchas from time to time. Sometimes they ignore our correct answers, and we have to solve dozens of captchas. Sometimes you get captchas like this. I’m probably not the first one to see this:

(Fahrrad is German for bicycle, but here it was probably shortened to “bike,” which can also mean any kind of motorcycle. In fact, this is more of a moped or scooter.)

What the actual FxxK

Yeah, and then we have this one. Story behind it: I have this Chromecast Ultra gathering dust for a couple of years now. Today I decided to give it a try. I connected it to my optical computer-to-human interface (the one firing a million protons directly through your eyeballs into your brain) and tried to register it. After the annoying fact that I have to install Google’s Home app to set up the Chromecast, I was referred to another app called “Device Utility.”

I have to install this tool to “unlock” my Chromecast. Some kind of protection? Seems like Google has activated a “kill switch” after a period of non-usage.

However, the problem is what you can see here: My Pixel 8 Pro – manufactured by Google – is not compatible with this app.

All I wanted was Netflix. Now I’ve ended up spinning up a virtual Android device in Android Studio.

Conclusion

People don’t get tired of announcing that AI will replace developers. Mostly salespeople. Microsoft claims that AI is already writing 30–50% of their code. They also admit, that quality of C+ is behind expectations. In 2030 they plan to reach a rate of 95% , though human oversight and authorship remains important. But they are silent about further details. I am not totally disagreeing, but I am pretty sure that there’s still a bumpy road ahead. You may argue: Well, then you don’t know how to properly operate AI. Problem usually sits in front of the screen, as they say. Fair enough. That’s why I want to share some thoughts and experiences.

AI is increasing my efficiency to a very high degree (I am referring to common LLM solutions here, but sometimes prefer the term AI, knowing that this can be misleading). But it also requires good skills as a developer. And if you don’t keep your head up, you lose more time “playing around” than you win by using AI.

Speaking of playing around: These days I was just doing that. AI enables me to do this more frequently. It’s so much easier to wrap your head around foreign ground. And it’s fun, without crawling through endless Google search results or SO threads. So I am having this idea of a simple side project, it’s one of those little things I get stuck on. And I was in need of an SVG showing a bike. I asked several LLM to generate one.

I am keeping it short. They all grandiosely failed.

Gemini does not support SVG creation on its web interface. Claude does. And this was the first result: (I am sparing you the detailed prompts, basically I just asked for a simple bike SVG, no extensive prompt, follow-up prompts were quite brief, but detailed enough to get the point across)

OK. Close. What else you got?

We are getting closer. But still not a bike.

No. Last try?

Ok. Close. Grok. Your turn.

Expressionism. ChatGPT. Last chance to hold the lance for AI.

What’s that? You need a doctor?

Nope. Next try!

Yeah, that’s impressive! But wait. That’s not SVG. Caught you cheating. Use it as a template, try again!

What? Last try. You can do it!

Cheating. Again.

OK. Unfair comparison. I know. This is clearly not about coding, but it makes a point. I am not blaming an LLM for not being able to create a simple bike SVG. Especially because SVG creation is not really something a developer does and it’s not what LLM are meant for in the first place. But it’s a perfect metaphor for how the process quickly goes sideways:

Sidenote: Recommended associated read: Simon Willison had a similar journey with Qwen at a much tinier scale, though.

Those bots have an over-confident way of approaching things. They always try to flatter the user (I learned this wonderful word “sycophancy” these days – and forgot it the next day). And they rotate around problems. Did you notice how ChatGPT gets back to the pixelated image after one unsuccessful try?

While they are perfect in solving “closed and recurring tasks”, they suffer from the “open-ended” problem. [Microsoft just published a study about jobs that can best benefit from AI](And most of them are exactly this: Translator and interpreters. Salespeople - easy, just train a model on product data and you are good. And so on. But a good developer has something that AI does not have (right now): Creativity and foresight.

(Don’t get me wrong, I don’t want to discredit the capabilities of those jobs!)

Another devastating example where Gemini-Pro failed on a large scale: I asked for a simple Python script to download an XML sitemap and parse a couple of URLs from it. Please trust me when I say I did the prompting correctly. Also, Python and XML are not exactly an exotic combination. However, after 1 hour, over 80 prompts, and I-don’t-know-how-many tokens, I gave up. Gemini wasn’t able to deliver (I am using gemini-cli for that). It kept circling around the same problem, ignoring correction advice, and repeatedly promising, “Now everything should work,” many, many times. If you want to see more interesting examples of the mixed quality of LLMs at coding, check out this wonderful video .

If you are into troubleshooting or exploring, you may have experienced this: You ask the AI to fix a problem or explain how to achieve something. It will provide you with a solution. Which is usually quite overwhelming. You start implementing this “pamphlet” from step 1 to step 42 or so and at step 2 you get stuck.

Why? The system you are working on returns an error. We have to rethink our solution process and feed that back to the LLM. We can’t blame the machine, because the reason is usually: You did not provide enough context about the environment and circumstances. But it’s also important to say: How do you know what details are missing?

You can’t. Especially in troubleshooting, it’s part of the process. The process!

The process of troubleshooting is not having a page of steps to follow and after 10 minutes you are done. If you are a system administrator, you know what troubleshooting means. It’s a complex forth and back, trial-and-error rocky road. You get better with time, that’s what a good, seasoned, senior system administrator is: Someone who has seen a lot of problems and knows how to approach them.

Sure: AI has seen a lot of problems, too. It’s literally in their “brains” as they are trained on a lot of problems. Still, what they haven’t learned yet is: guiding you through processes step by step. Or at least offering it! Instead: wasting 1,000 tokens on a 42-ish-step solution.

You may think it’s part of the business model: Why should the bot efficiently guide you through a process, if it can provide you with a 42-ish-step solution, which won’t work and then: Your context window is out of tokens. Purchase the pro plan or wait 6 hours to get this thing done.

I asked ChatGPT to look over my article, do a fact check. It claimed that this part sounds to conspicious and I should mention that this is due to model design and use, not malice. So, disclaimer: My goal is not to imply a business model behind this phenomenon. I’m just kidding, okay?

Wich finally leads me to more than ranting and complaining with this word of advice: End every prompt like this. It will save you a lot of time and tokens:

Guide me through the process step by step, don’t answer all at once, wait for my confirmation and feedback before you continue.

But what about coding, at all? There are so many platforms offering perfect low-code or even no-code solutions. People telling you they built whole businesses without writing a single line of code.

Don’t believe it? You tried it yourself? Try.

I am not saying that AI/LLM are not capable of e.g. writing code. They are - totally, unironically!

But “real apps” are complex. They are not just an HTML one-pager with a nice parallax effect, some neat JavaScript interactions and CSS. There are backends, data models, APIs, authentication, security, data storage and so on.

These days I was writing this little app that ought to visualize biking trips on a map. It grew a little out of hand. Starting with Python in a Jupyter notebook, I ended up with a full-blown web app. And you know what? I did it almost the “vibe coding” way. See? I am also one of those guys with this impressive stories! I only spent time thinking about user experience and features.

And at some point I thought: “Hey buddy, what did you actually do there under the hood?”

Yes. AI was able to build this complex app (FWIW: It’s Claude 4 that right now provides best results for coding - not being paid for that).

I am using an extensive pre-prompt instructions file , telling the agent to use a structure of separated files, speaking function names and so on. Still I was facing a huge code base after some hours of “vibe coding”. The main app file was over 5,000 lines of code. I reached a limit, where even the agent wasn’t really able to process all the context. It was a mess. Ok, blame me. My instructions probably weren’t that precise. “The problem sits in front of the screen”, as they say.

Well, we still can use the agent to refactor, can’t we?

Yes, we can. I did it. And it didn’t really work out. It was creating stubs and redundant functions and endpoints.

The end of the story: I had to crawl through the code myself. Like back in the days. And it was not a difficult thing, I just spent two or three hours or so, strategically scraped through the codebase, identified redundant code, moved it out of the main function, cleaned up file structure. Like back in the days. Sure, with the help of AI, but this time only as a sidekick.

And this is what I am saying: You still need to be a good developer to understand code architecture and how code works. AI will not be able to deliver a fully fledged market-ready app for you. Not now.

However, this is not only a rant or my two cents, it’s also about “things I learned”, so let’s summarize:

Seven Rules of AIngagement

1. Always include something like “step by step” in your prompts if you are about troubleshooting or exploring (always means: “if applicalble”)
2. Use a pre-prompt instructions file to guide the LLM/Coding Agent!
3. Split the work! Use one agent to get the pre-prompt instructions that you feed to another agent instance, let another agent build you the first starting prompt, and so on - I’ve established a mixed toolset, Copilot on one hand, Gemini-CLI on the other and some browser-based agents on the third hand
4. I also learned that it is really helpful when the agent maintains a “protocol” that contains the initial task and then, in very short bullet points, the current status of the task – this way you can always ask the agent to continue with the last task without having to re-explain everything
5. And this may not bring me many fans: Also separate tasks and codebases, to keep the context window small. This is the rise of some kind of microservices-ish architecture, because “monolithic” apps can’t be eaten by the agent.
6. Discipline when it comes to version management. Let the LLM fix a thing, then commit the changes. Fix. Commit. Fix. Commit. You catch my drift.
7. Add a “Got it? Any Questions?” at the end of your prompt! You would be surprised how this changes whether the agent starts right away (remember: Sycophancy!) or gets back to you with some surprisingly important questions!

What about “non coding”?

Yeah, ok, this is a sweeping blow. I also have a opinion here.

Do you remember, back in the days, when you were “Googling” for information? First people started looking for the whole term, like “how to build a web app with AI”. But the Google algorithm was improving and so did the prompts. People reduced their search terms to “web app AI”. You don’t necessarily need filler words to understand the intention of a search. And not only filler words, also spelling or grammar.

Right now the same happens to AI, apparently. Sometimes I catch myself typing prompts with so many errors and grammar mistakes. But it’s doing its job. I wonder if this leads to “degradation” or better “optimization” of our language. Have you heard of Gibberlink or Neuralese ? They are supposed to be “languages” or “concepts” that are used by AI to communicate with each other. Like more efficient versions of human languages, without all the noise and “unnecessary” words. As we are talking to AI more and more, what if, at some point, we adapt our language to the AI?

Second thing that will change is probably how we will use our “computers”. There are not many user interfaces right now. We have the GUI - the graphical user interface, it’s the colorful windows and buttons you see on the screen. You can touch them or use your mouse cursor to select text. We have the TUI - text user interface, which is most common for CLI (command line interface) applications. You can type commands and see the output in text form. And we have the VUI - voice user interface, which is becoming more and more common with the rise of smart speakers and voice assistants. All of them have their strengths and weaknesses. Building a PowerPoint app from the command line is probably quite difficult (side note: it’s possible and it’s fun to automate data analytics pipelines using Python libraries and create extensive PowerPoint slide sets!). Grepping through multiple servers’ request logs using voice or a mouse? No one does it.

(Leaving out [brain computer interfaces](here, because they are still in the research phase and not widely used yet.)

There are exceptions, though. Do you know those movies where characters start typing commands to do simple tasks, like zooming in? They have the most expensive and advanced technologies, but somehow forgot to add a “Zoom” button, so they have to program this rare feature in real time—without even having a shell open. Absolute geniuses!

Right now, we are using AI in two ways: Most common ways are TUI and VUI. And this is interesting. Because in most cases this breaks the flow of work. You have to write long prompts, edit them, add more details. And after 5 minutes your context window is so full, you start all over again. Then you have to copy results back, interpret them, translate them into “graphical” interactions. And so on. Doesn’t really sound like the peak of possible efficiency, does it?

(Good thing: You are forced to think about what you want to achieve. Which should be a no-brainer (even pre-AI): Think before you act! With AI it’s even more important.)

Well, here comes the MCP into play - the Model Context Protocol, which connects the agent to other systems, like - sticking with this example - PowerPoint. You prompt your task, the AI controls PowerPoint and builds stuff. But at the end, you still need to read, type or speak your commands.

What we are missing is a GUI for AI. And as long as we don’t have brain computer interfaces, we will find a way around this. Microsoft is trying to achieve a similar thing with their Recall feature , which raised a lot of anger and concerns about data privacy - which is totally understandable. Recording your screen, processing it on some server at some unknown location? Seriously?

But this will be the future. Because this allows reactive graphical user interfaces and I am not saying dynamic ones. Thinking of a user interface that changes because of some previous actions - that doesn’t sound very useful, right? Imagine your car changes the functions of your brake and accelerator pedal. It’s more an interface that “follows” the user, not makes the user follow the interface. Form follows function.

Sounds bold, I know. But do you really think that we will still push tons of words into browser windows in 10, 20 years?

Monitoring SSL Traffic of Android Apps is quite a challenge these days. While it’s an important way to understand how Apps work, find security flaws or possible data breaches, a weak encrypted traffic also is a potential security risk.

These days, most apps use a thing called “certificate pinning”, which makes it quite difficult to monitor traffic. You need a rooted device, to bypass certificate pinning. However, this leads to a cascade of challenges and you easily get sucked into a rabbit hole with obstacles, turning points and furstrating dead ends.

“Lukily there’s AI guiding me through the process!” You may think. Well. No.

Gemini requires arm-twisting to make you help:

I cannot provide a step-by-step guide on how to bypass certi"ficate pinning on a rooted Android device for malicious or unethical purposes. My purpose is to be helpful and harmless, and assisting in such activities falls outside of my ethical guidelines.

ChatpGPT is a little too helpful. it’s providing you with a overwhelming large emoticion spiked list of 20-or-so steps. Usually you stuck at step 5 and there the chaos begins (like this article, which is a bit contradictory, right?).

Note of advice: That’s common behavior for most fo the agents. In cases like that, your prompt should always contain an instruct ion like" Guide me through the process stepy by step, dont answer all at once, but wait for my confirmation to continue."

Preperations

Let’s assume we are working on Window and having Android Studio up and running, including the propper Android SDK. So lets start with setting up a virtual Android device wihtouth Google Play. Google Play-enabled devices cannot easly be rooted. Let’s drive with the Pixel 6 Pro and Intel x86_64 (you may choose a different image, just make sure it matches you host’s hardware setup).

App Installation

As we dont have the Google Play store at hand, we need to find way around. There are two options. The easiest one: Use a service/plattform like apkmirror.com or uptodown.com. I prefer the first one, as uptodown comes with a wrapper that - once installed on the device - installs the actual APK.

The second way is to use a device that has the app installed, either a virtual one or your actual phone, and pull the APK from there.

We will enable developing mode and USB debugging on our source device and connnect it to our host system. Using allmighty Android Debugging Bridge adb we can pull the APK from the device. First make sure your device is connected and recognized by adb:

1cd /AndroidSDK/platform-tools # your Androd SDK may be in a different location!
2adb devices # to check if your phone is connected
3adb shell list pm packages -f | grep Foobar # to list all installed apps with their APK path and filter by your app's name
4adb pull /data/app/com.Foobar/base.apk # to pull the APK from your device

These days most apps are splitted into different APK files (for different purposes like screen size, language, etc.). We need to pull all APK’s. Let’s first find out the APK path of our app:

1adb shell pm path de.foobar.app.mobile

This will probably return a list like

1package:/data/app/de.foobar.app.mobile-1/base.apk
2package:/data/app/de.foobar.app.mobile-1/split_config.en.apk
3package:/data/app/de.foobar.app.mobile-1/split_config.xxhdpi.apk

We need to ´adb pull` all of them one by one.

1adb install -r de.foobar.app.mobile # if we have one APK only
2adb install-multiple -r base.apk split_config.en.apk split_config.xxhdpi.apk # if we have multiple APKs

That’s it!

Rooting the Virtual Device

Now lets prepare your virtual device, which is usually a quick step. First, apparantly, disconnect your source device from your host. Then start the virtual device in Android Studio:

1cd /AndroidSDK/emulator
2emulator -list-avds # optional: list all available virtual devices
3emulator -avd AndroidRoot # that's the name of the virtual device, we created earlier

This will take some seconds. Once done, we need to make sure the device is rooted:

1adb root

We can test the success of this command by checking the device’s shell:

1adb shell

On the device check the output of these commands:

We are in! For our monitoring we need to enable the proxy on the device. This is done by setting the proxy settings in the device’s network settings.

1adb shell settings put global http_proxy 10.0.2.2:8080

(You may reset the proxy setting with adb shell settings put global http_proxy :0)

Building the monitoring stack

Here comes the challenging part. Here’s the rough process:

We will need four different terminal windows

• one is running the Android emulator (if you start it from the console)
• one is running the monitoring tool (MITMProxy) on the host
• one is running the Frida server on the device
• one is running the Frida tool on the host, injecting a script into the app

Installing and running the monitoring tool

This is the easiest part. We use MITMProxy to listen to the HTTP traffic coming from the device. We can use the Python’ic way, if you prefer you may use brew or the installer form their website.

1pip install mitmproxy

Done. Now run it and see what happens:

1mitmproxy --listen-port 8080

If you’re happy, you already see HTTP traffic coming through and probably some warnings on the bottom, telling you that - of course - https traffic is not decryptable.

(Screenshot already shows HTTPS traffic, but you will see only HTTP traffic at first)

Installing and running the Frida server

Now we will install the Frida Server on the device. Frida - in short - is a tooblxo for developers to debug or monitor apps. You can get the latest version from their GitHub repo . Make sure to download the server package for Android and the correct chipset, like in my case frida-server-17.2.14-android-x86_64.xz. Extract its content which will get you a binary file that you push to the virtual android device:

1adb push frida-server-17.2.14-android-x86_64 /data/local/tmp/frida-server

Make it executable:

1adb shell chmod 755 /data/local/tmp/frida-server

No shell into the device and run the server (the & at the end makes it running in the background):

1adb shell
2emu64xa:/data/local/tmp/frida-server &

Injecting “The Script”

We also need to install the Frida-toolbox on our host machine. There are several ways, we stick with the Python’ic-way:

1pip install frida-tools

Not much to explain here. After you done, we need to find a script that we “inject”. This script will circumvent the certificate pinning of the app. There are many scripts out in the wild and you can even ask AI to write you one. I am using this one:

` https://gist.github.com/akabe1/5632cbc1cd49f0237cbd0a93bc8e4452``

Download it to a nicer name like frida_multiple_unpinning.js.

Now we tell Frida on our host to run the app we installed earlier and inject th unpinnign script:

1frida -U -f com.Foobar.apk -l frida_multiple_unpinning.js

That’s it! Now switch to your terminal window running the MITMProxy and you should see a lot of traffic coming through!

Some times, some apps, some configuration will see that there’s something going on with their traffic. Sometimes the script will fail, the app will crash or not work at all. You need a little bit of trial and error to find the right unpinning script for your app.

Besides, you can pre-compile your script, which sometimes helps to avoid detection.

1frida-compile frida_multiple_unpinning.js -o compiled.py
2frida -U -f com.Foobar.apk -l compiled.py

Round up

• make sure you are root
• install the APK on the device using adb
• start the Frida server on the device
• start the proxy on the host
• optional: use adb
• start the app on the device using Frida and inject the script (make sure the app is not running, or you get error Permission denied)

 1cd /AndroidSDK/emulator
 2emulator -avd AndroidRoot
 3cd /AndroidSDK/platform-tools
 4adb root
 5adb install -r com.yourApp.apk
 6mitmproxy --listen-port 8080
 7adb shell
 8$ /data/local/tmp/frida-server &
 9# back on the host
10adb shell pm list packages -3
11frida -U -f com.yourApp -l frida_multiple_unpinning.js

naming convention

Global

Use a category and naming convention that allows you to track variables, metrics and dimension from the website over the tag manger to the backend of your analytics solution and the dashboard. Every category is also assigned a particular numeric range. This will improve your debugging and troubleshooting process.

The following basic categories should represent your default setup.

• Page related information (for a single page view, from 1 to 19)
• Site related information (for the whole website, from 020 to 39)
• User related information (describing the visitor, from 40 to 59)
• Misc information (meta information that usually is not part of an analysis or cannot be assigned to the other categories, from 59 to

Data Layer

 1	datalayer = {
 2		page: {
 3			category: 
 4			name: 
 5			id:
 6			date:
 7			language:
 8		},
 9		site: {
10			provider:
11			project:
12			domain:
13		},
14		user: {
15			status: [anonmyous, logged-in],
16			behavior: [returned, new],
17			language:
18		},
19		misc: {
20			timestamp:
21			debug: 
22			utm: 
23			interaction_details:
24		}
25	}

“I’ve implemented a page.category attribute, which I find invaluable. I’m surprised so many implementations omit it. It allows for quicker traffic analysis at a higher level than individual pages. Categories help maintain order as pages inevitably become disorganized over time.”

Data Elements

Your Data Element name should consist the numeric ID referring to the ID of your (Adobe) Analytics-implementation followed by a separator (like the pipe symbol |) and the address within the Data Layer object. Left pad the ID to a length of three to not mix up the “order by name” feature. The second part is the exact address of the element within the data layer. If this element does not correspond to the Data Layer, use a short description.

• 000 | datalayer.page.name
• 010 | datalayer.misc.utm

Dimensions

eVars, props and events are identified by a short description build from the category and the name of the information, followed by the name and the kind of variable (t for traffic variables aka props, c vor conversion variables aka eVars and e for event):

• Page Name (t010)
• User Status (c020)

Events

Naming Rules

You wil need three categories of rules, based on the action the perform

• Page View rule (mostly just one, will trigger a page view)
• Click rules (will trigger click events)
• Misc rule (will not trigger any call, but prepare tracking or add event listeners)

Besides that, you will have three main attributes for page view and click rules that should be available in the component name itself:

• the according event id (custom success event, even page views can have an event ID)
• a brief description
• how a rule will be triggered (Page Top, Document Ready, Click, Direct Call, …)

Name your rules using those three information (I prefer the pipe symbol as a separator)

[TYPE]|[TRIGGER]|[EVENT ID]|[BRIEF DESCRIPTION]

If your rule covers more than one event, post them all as a comma separated list (50,51,52). If there are too many rules, you may overthink your strategy.

For example:

• PV | DocReady | 10 | All Pages
• PV | DocReady | 20 | Search Pages
• CL | Trigger | 50,51,52 | Click
• CL | Click | 60 | Bookmark Page
• MISC | Helper Functions

Reason

In the rules overview this helps you to get a good first impression of all existing rules and it allows you to quickly filter the list for wanted rule types. Also, if you lookup a particular event, you quickly see the particular component that somehow works for this event.

Naming Components

A component can be a trigger/event, condition or action. When you create a component, Adobe suggests a default name. Never use this name. The name should start with the exact sequence you defined for the rule name.

For example:

You have a rule to trigger page views:

PV | DocReady | 10 | All Pages

This is how you name the components of the rule

• trigger/event: PV | DocReady | 10 | All Pages | page top
• condition: PV | DocReady | 10 | All Pages | user is logged in
• action: PV | DocReady | 10 | All Pages | set variables
• action: PV | DocReady | 10 | All Pages | send beacon
• action: PV | DocReady | 10 | All Pages | clear variables

Reason

If you look up a particular event ID, you quickly see all according components. If you look up particular variables that belong to different events, you know quickly identify, where this variable is being used, without clicking into the component itself.

Error Handling

If you have custom code, you should always implement try/catch error handling that also refers to the parent rule:

1let rule_name = '`PV | DocReady | 10 | All Pages';
2try {
3
4
5} catch (e) {
6
7	_satellite.logger.error(rule_name + ' | ' + e);
8
9}

If your code grows and get more complex, you should and can, add multiple try/catch lines into your code. In this case you should add a brief description of the current area to the error message:

 1let rule_name = '`PV | DocReady | 10 | All Pages';
 2try {
 3
 4
 5
 6} catch (e) {
 7
 8	_satellite.logger.error(rule_name + ' | init something | ' + e);
 9
10}
11
12try {
13
14
15} catch (e) {
16
17	_satellite.logger.error(rule_name + ' | calculate another metric | ' + e);
18
19}

Reason

Now, if you face errors in the debug console, it’s easier to find the source, even if you handle long code blocks. You don’t need to click through the stack trace.

Logging and Documentation

The same goes for logging. Always logg what you are doing. I don’t need to mention, how important inline documentation is. ;) Logging works the same as error handling:

1_satellite.logger.info(rule_name + ' | calculate another metric');

Reason

Now, if you filter the browser console, it’s easier to get rid of the noise and debug a problem.

Inactive Rules or Data Elements

You will and cannot avoid to have inactive/disabled rules. You should keep that amount low, because they pollute your library. If you have inactive rules, add a prefix “INACTIVE” to the rules name and, preferably also to it’s components.

Reason

If you look up code parts, data elements or anything, the search will return you rule and component names. From those component names you will not see, if a result belongs to an inactive rule. The prefix INACTIVE will help you to quickly identify those results.

Shortcuts

First of all, I strongly suggest adding some custom shortcuts because the default ones are a bit unintuitive. For example, CTRL + SHIFT + ' gets you to the terminal, while SHIFT + FN + F6 brings you back to the editor, andSHIFT + B opens the sidebar. What a mess! A more consistent setup can definitely improve your workflow.

To customize your shortcuts, press CMD + K followed by CMD + S or look up “Keybindings” in the command palette (SHIFT + CMD + P). There’s a small icon on the right to open the actual JSON file, which makes configuration much easier. What about jumpin between the sidebar, editor, and terminal using the arrow keys while holding CTRL + SHIFT?

 1[
 2    {
 3        "key": "ctrl+shift+down",
 4        "command": "workbench.action.terminal.focus"
 5    },
 6    {
 7        "key": "ctrl+shift+up",
 8        "command": "workbench.action.focusActiveEditorGroup"
 9    },
10    {
11        "key": "ctrl+shift+left",
12        "command": "workbench.action.focusActivityBar"
13    }
14]

Add more shortcuts to make your mouse-less life easier. But thats not all, I also suggest to use a tool like Espanso ! It’s a very powerfull text expander with two main interesting features:

• if you start typing certin chars, it will automaticall expand them to something you configured
• it also provides a quick search feature that you can trigger with OPTION + SPACE

So here’s my strategy, let’s take some Python commands for example. I am not using unique trigger keywords, but the same one to define particular categories ( see a full example at this Gist )

1matches:
2  - trigger: "#p1"
3    replace: "source .venv/bin/activate"
4  - trigger: "#p"
5    replace: "python3 -m venv ./.venv"
6  - trigger: "#p"
7    replace: "pip3 install -r requirements.txt"
8  - trigger: "#p"
9    replace: "pip3 freeze > requirements.tx"

When I open the quick search bar, I type #p to get all Python commands. Ofcourse you could also add unique trigger keywords, I prefer it like that because I don’t remember all those keywords anywawys. Now I can just show quickly browse through commands for a given category, like JavaScript, Python, C# - you name it!

Containers

Warning: Requires Docker!

Next, let’s talk about containers! I strongly recommend using containers for your projects. Not all of them, of course, sometimes a virtual environment is totally sufficient. But containers! Oh boy! They are quick, easy, and help keep your system clean. VS Code allows you to set up dev containers—a closed ecosystem where you can configure your development environment. This is especially useful when switching between different languages, frameworks, or systems.

For example, if you want to test Ruby without installing all the dependencies on your system, create a .devcontainer folder within your workspace and place two files in it: devcontainer.json and Dockerfile.

Example Setup for Ruby:

Dockerfile

 1FROM ruby:3.2
 2ENV LANG=C.UTF-8
 3ENV DEBIAN_FRONTEND=noninteractive
 4RUN apt-get update && apt-get install -y build-essential git curl zlib1g-dev \
 5    libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 nodejs && \
 6    rm -rf /var/lib/apt/lists/*
 7ARG USERNAME=vscode
 8ARG USER_UID=1000
 9ARG USER_GID=$USER_UID
10RUN groupadd --gid $USER_GID $USERNAME
11RUN useradd --uid $USER_UID --gid $USER_GID -m $USERNAME
12USER $USERNAME
13WORKDIR /workspace
14RUN gem install bundler
15EXPOSE 3000
16CMD ["irb"]

devcontainer.json

 1{
 2    "name": "Ruby Development",
 3    "build": {
 4        "dockerfile": "Dockerfile",
 5        "args": {
 6            "USER_GID": "1000",
 7            "USER_UID": "1000",
 8            "USERNAME": "vscode"
 9        }
10    },
11    "settings": {
12        "terminal.integrated.defaultProfile.linux": "bash"
13    },
14    "extensions": [
15        "rebornix.ruby",
16        "castwide.solargraph"
17    ],
18    "postCreateCommand": "bundle install",
19    "remoteUser": "vscode"
20}

Save everything, and VS Code will prompt you to create and open the dev container—no need to manage Docker manually! If you don’t see the message, open the “Remote” container manually by clicking the blue icon in the bottom left corner or using the shortcut OPT + COMMAND + O.

Git

You don’t necessarily need a GitHub account to store repositories remotely, but I encourage initializing a repository locally. This helps you track changes and maintain your work efficiently.

Configuring Multiple GitHub Accounts

If you use different GitHub accounts (e.g., for personal and professional projects), modify your Git config file:

1[user]
2    name = John Doe
3    email = john.doe@example.com

Branching Strategy

If you don’t use Git daily, consider a simple branch strategy:

1. Use the main branch for stable code.
2. Use a debug branch for hotfixes or minor fixes.
3. Use a dev branch for new features.

Example Workflow:

1git checkout dev
2git pull origin main
3# Make changes
4git commit -m "Your commit message"
5git push origin dev

Once you’re ready to merge changes:

1git checkout main
2git merge dev
3git push origin main

This keeps your codebase organized and reduces conflicts. If needed, Copilot can assist with Git commands!

Copilot

Copilot enhances your workflow with:

• Inline code completion
• Inline chat (CMD + I)
• An editor that handles multiple files (CMD + SHIFT + I)
• General assistance via chat
• Commit message suggestions

Improving Copilot’s Effectiveness

To help Copilot assist you better:

• Use meaningful function names.
• Add inline comments.
• Write clean, structured code.

Customizing Copilot

You can fine-tune Copilot’s behavior by modifying your workspace settings. Create a settings.json file inside the .vscode folder and add custom configurations.

Example Settings:

1"github.copilot.chat.localeOverride": "fr",
2"github.copilot.chat.commitMessageGeneration.instructions": [
3    {"instruction": "Start every commit message with FOOBAR"}
4],
5"github.copilot.chat.codeGeneration.useInstructionFiles": true

Now, create a ./.github/copilot-instructions.md file with your custom instructions. Example:

1- My project goal is a simple and performant API interface.
2- Code must follow modern coding standards.
3- Functions should have a maximum of 20 lines.
4- No redundant functions.
5- Each function must have a corresponding test function.
6- Include example data for testing.
7- Use multiple files, grouping functions logically.
8- Prefer class-based structures over standalone functions.

Finding More Rule Sets

For additional guidance, check these resources:

• Awesome Cursor Rules
• Mawla Cursor Rules
• Instructa AI Prompts

Claude, ChatGPT, Gemini, and Probably Even v0!

It doesn’t end with Copilot. And again: You can achieve a lot with free plans! But if you really want to improve your process and build big things, consider using LLMs.

Before diving in, heads up: Don’t make the same mistake as this poor guy :

Guys, I’m under attack ever since I started sharing how I built my SaaS using Cursor. Random things are happening—maxed-out usage on API keys, people bypassing the subscription, creating random stuff in the database.

Don’t expect an LLM to generate a perfect app if you just throw some poorly written sentences at it. Garbage in, garbage out. So, what should you do instead? Chain it!, baby.

• Start simple.
• Iterate.
• Split your project into independent parts.

Think about developing an Android App for example. How would a human developer approach the task, what information would be required?

1. The overall concept: What is the app about? What language is it written in? Let’s call it the “soul”
2. The UI/UX concept: How should it look and feel? The “body”!
3. And, to stick to the image: The “viscera”: The architecture, functions, features and internal structure.

Now, ask ChatGPT to consider this “trinity” and generate three structured pre-prompt instructions for you.

1. In Copilot, use the soul prompt in your pre-prompt instructions, as mentioned above.
2. Then, feed the body and viscera prompts into Copilot, Claude, or v0 to refine the results further.

Of course, this isn’t a cut into stone,but it should give you a structured approach to leveraging AI effectively.

But do not forget this: AI Is a Tool, Not a Magic Wand. Don’t take AI-generated results for granted. The golden rule from the early internet days still applies:

“A computer is only as smart as the person using it.”

Always question and understand the output. Don’t expect AI to perfectly execute any task instantly and without errors. Because: The developer is still you.

In recent years, I’ve worked in and for a variety of companies, ranging from hands-on SMEs, through more or less organized startups, to large corporations and agencies. I mention this not to brag, but to illustrate the breadth on which my following observations are based.

The larger the company, the less capable it is of developing innovative ideas, realizing them, or optimizing itself and its processes. (This is not a universal statement, but certainly a tendency.)

Why is that? So far, I’ve identified two important reasons:

1. Bureaucracy, Compliance & Processes

The larger the company, the less individual the processes. My favorite example is “Code of Conduct” courses. These typically cover four illustrious topics: Social Ethics, Business Ethics, Data Protection, IT Security. First Aid and Fire Safety are also often included. There’s nothing wrong with the content; everyone should “attend” these courses at least once. But the content is often very broad. For example, “Bribery” is a popular topic, but it undoubtedly serves a limited audience. Not to mention the impact: If I want to accept a bribe, the corresponding course certainly won’t stop me.

Nevertheless, everyone must deal with it. The participation requirement is often, if not always, distributed using the watering can principle: They are part of onboarding; must be repeated once a year. If you switch to another agency, you take part in such courses again. If you switch clients within the agency, the new client requires participation in their “own” courses - which are, of course, identical in content.

The result: Frustration, boredom, and unnecessary costs. The companies’ reaction: They spend a lot of money to pack the content into “modern” formats, conveying it playfully or in “stage plays”.

Why isn’t there a Code of Conduct driver’s license, a certificate? A test tailored to my field, which I take once a year and can show if needed!

But it’s also the processes in daily business that stifle innovation. I once worked for a large German employer, and we wanted to set up a service to simplify the import of CSV files. Technically, it wasn’t complicated; the files were small, and it was just about adjusting column names. However, we needed a service, a small virtual server or Docker container running in the background. It took months to find out which department could help us. Ultimately, we had to stop the project without success. A hurdle that would be unthinkable in a startup.

Recently, I stumbled upon a very nice talk on the Skunk Works principle, explaining its origin and impact. The idea is, roughly summarized: A small team of experts works independently and as free from corporate constraints as possible on a project.

Another solution would be to not only allow but actively promote U-boat or pet projects with the necessary infrastructure. Shortly after my studies, I worked at a somewhat larger “startup” and eventually implemented a technically simple search mask that massively facilitated the team’s work. Needless to say, my team lead wasn’t very happy about the “misallocated” hours.

2. Turf Protection, Tunnel Vision

This is a phenomenon I’ve observed many times; I call it the “wrong Gallic village”. A Gallic village that defends itself against invaders can be a good thing. But what if it’s not invaders but innovators? New ideas are either quickly perceived as negative criticism of the previous quality, or there’s simply fear of losing one’s established position. A Gallic village that resists any outside influence has detached itself from the company’s goal.

Existing solutions are rarely questioned. This is a big problem, especially in the technological field, where new possibilities arise weekly. Three arguments are then used to protect against the impending invasion innovation:

1. Never change a running system.
2. No capacity.
3. Too complicated.

I’m not saying that the motto “Never change a running system” is fundamentally wrong. You don’t have to jump on every bandwagon. But eventually, the last train has left the station. And before you know it, the solution is based on an outdated technical platform that is no longer supported or mastered by anyone. Software development, in particular, suffers from this phenomenon: A program is never finished.

“No capacity” is an excuse that shifts the current, perhaps unknown problem into the future. But this shift is not linear. If you shift every adjustment into the future for ten years, they don’t spread over a period of ten years in the future. They accumulate at the point where the future begins. And that’s tomorrow.

“Too complicated” is an argument that undermines itself. If adjustments to the current solution are too complicated, we should invest time sooner rather than later. Because software development also suffers from this: A program rarely becomes less complex over time; rather, the opposite is true.

The art is to have an agnostic attitude towards change without succumbing to it.

Nicky Reinert
Institute for Digital Challenges
E-Mail: nісkуrеіnеrt [at] gmаіl [dot] соm
Characters obfuscated for spam protection, please type manually

Note: The “Institute for Digital Challenges” is not a registered institution, but a private research and analysis project run by an individual as a hobby.

Responsible for content according to § 55 Abs. 2 RStV

Nicky Reinert

Lichtenbergstr. 40 10243 Berlin

Disclaimer

Liability for content

The contents of this website have been created with the utmost care. However, no guarantee can be given for the correctness, completeness, and topicality of the content.

Liability for links

This website contains links to external third-party websites, over whose content no influence is held. The respective provider or operator of the linked pages is always responsible for their content.

Copyright

The content and works on these pages created by the site operator are subject to German copyright law. Reproduction, processing, distribution, and any kind of use beyond the limits of copyright law require the written consent of the respective author or creator.

Controller

Institute for Digital Challenges
E-Mail: nісkуrеіnеrt [at] gmаіl [dot] соm
Characters obfuscated for spam protection, please type manually

Note: The “Institute for Digital Challenges” is not a registered institution, but a private research and analysis project run by an individual as a hobby.

Type and Scope of Data Processing

Our activities include research, data analysis, data engineering, and data mining. We generally do not process personal data as defined in Art. 4(1) GDPR. If, in exceptional cases, personal data is provided to us in the context of projects, it is anonymized before any processing, preventing any link to identifiable individuals. We do not share data with third parties, nor do we analyze personal data using external services.

Task Manager App “Procrastino”

Our mobile app enables synchronization of tasks with Google Tasks. Signing in with a Google account is required for this purpose.

Processed data: Only the information required for synchronization (e.g., task contents, task metadata) is processed.

Storage: All data is stored exclusively on your device in the app’s storage. No storage or processing takes place on servers operated by us.

Disclosure: No data is shared with third parties.

Advertising/Tracking: The app does not contain any advertisements and does not use analytics tools.

Google services: The use of Google Tasks is additionally subject to Google’s Privacy Policy: https://policies.google.com/privacy

Hosting via GitHub Pages

Our website is hosted via GitHub Pages (GitHub Inc., 88 Colin P Kelly Jr St, San Francisco, CA 94107, USA; a company of Microsoft Corporation). When accessing the website, GitHub automatically processes technical information (e.g., IP address, browser type, operating system, referrer URL, timestamp) in server log files to ensure operation and security. For further details, please refer to GitHub’s Privacy Statement: GitHub Privacy Statement . We do not analyze these log files ourselves.

Legal Basis

Since we do not process personal data ourselves, our activities are not based on consent or legitimate interest. The logging carried out by GitHub is based on Art. 6(1)(f) GDPR (legitimate interest in operation, security, and stability of web services) by GitHub.

Rights of Data Subjects

Regardless of the above, you are entitled – where applicable – to the rights under Art. 15–22 GDPR (access, rectification, erasure, restriction, objection, data portability). For requests concerning processing carried out by us, please contact us at the above address. For the logging performed by GitHub, please contact GitHub directly.

Data Security

We take appropriate technical and organizational measures to protect the confidentiality, integrity, and availability of the data we use.

Cookies, Tracking, and Embedded Content

We use no cookies, perform no tracking, and embed no external content (e.g., web fonts, videos, scripts).

Changes to this Privacy Policy

We reserve the right to update this privacy policy to reflect changes in legal or technical requirements.

Effective date: 01 January 2025

The first time I read through the riddle, I was like: “Phew, no way, I’m going to skip this!” I looked at the example and the description, and it didn’t make sense at all. But when I’m stuck like that at the very beginning, the best approach to get a grip is: Visualize! (A concept I didn’t learn from [Daniel Bourke’s course about machine learning](but he perfected it!)

However, I thought it might be a good idea to use Ruby this time — my first time with this language. I won’t bother you with the environment setup. Again, I’m just using a containerized environment, which makes things so much easier.

Solution to the First Part

So, what is this about? It’s about finding siblings on a surface, connecting them with a straight line, and then extending this line to find two symmetric points. The total number of all those points forms the solution to this riddle. The green and red cells represent two categories of siblings on the map, and the blue cells represent the symmetric points (called antinodes) that we need to find. If you connect them (with red lines), you’ll see how all of them pass through two cells of the same color:

Looks difficult, right? Don’t tell me, it took me a while to understand the “gibberish” of the riddle – no offense, riddle master.

But wait until we add row and column numbers to the map and focus on just one group:

Now it seems quite easy: We just need to somehow identify the row and column distance between two siblings and use that to place the corresponding antinode.

Let me just throw my solution into the ring. It’s not the most elegant, but it gets the job done:

• First, I use two loops to find two siblings.
• Then, I calculate the row and column indexes for both siblings.
• From there, I can calculate the row and column indexes for both antinodes.
• Finally, I just check if they are within the boundaries before adding them to a dictionary.

 1from_char_array.each_with_index do |from_char, from_index|
 2    next if from_char == "."
 3
 4    from_char_array[(from_index + 1)..-1].each_with_index do |to_char, to_index|
 5        next unless from_char == to_char
 6
 7        from_row = (from_index / map_height).floor
 8        from_col = from_index - (from_row * map_width)
 9
10        to_row = ((from_index + to_index + 1) / map_height).floor
11        to_col = (from_index + to_index + 1) - (to_row * map_width)
12
13        puts "Found couple of #{from_char} at #{from_row}/#{from_col} and at #{to_row}/#{to_col}"
14
15        first_antinode_row = from_row - (to_row - from_row)
16        second_antinode_row = to_row + (to_row - from_row)
17
18        first_antinode_col = from_col + (from_col - to_col)
19        second_antinode_col = to_col + (to_col - from_col)
20
21        if first_antinode_row.between?(0, map_height - 1) && first_antinode_col.between?(0, map_width - 1)
22            antinodes["#{first_antinode_row}/#{first_antinode_col}"] = "#"
23        end
24        
25        if second_antinode_row.between?(0, map_height - 1) && second_antinode_col.between?(0, map_width - 1)
26            antinodes["#{second_antinode_row}/#{second_antinode_col}"] = "#"
27        end
28        
29    end
30end

If you’re stuck on this one, no worries: I have to admit, although it’s just roughly 20 lines of simple code, it took me a while to initially understand the task and then put everything together, because I was mixing up 0-based indexing, column calculation, and so on.

And because I am already 4 days behind, I didn’t even read the second part and continued to day 9…

What’s up, Ruby?

Rating: 10/12 – underestimated diamond

I was honestly surprised at how easy it is to get something done in Ruby. It seems as simple and straightforward as Python, and if you’re looking for a quick ’n’ dirty scripting language, I’d say Ruby is definitely worth a try.

This is the kind of task I really enjoy: finding algorithms for simple mathematical challenges! Let’s go—this time in… C++! (Phew!)

You’ll find my .devcontainer and Dockerfile on GitHub. Once again, I don’t want to bloat my system with additional compiler setups. It turns out, this is working like a charm.

Solution to the First Part

Well, the idea is quite easy: We got a list of operands and need to find the correct combination of + and * to get a target result. What we basically need to do is: Loop through all given equations, get all possible permutations of + and * for n given operands, and then do the math.

The first tricky thing here is: They are providing a lot of huge numbers. So using 32-bit integers isn’t sufficient. You not only need 64-bit integers, but also unsigned to work with numbers like 175.033.602.122.656.

But they also hid a little pitfall in the list of possible equations: There is one (exactly one!) duplicate entry. Which makes it hard to find. Would it be ten, 42, or 100 duplicate entries, you would somehow stumble over it. But now it took me a while, and I was close to throwing this task away. I started debugging and I wondered why I am only processing 849 equations instead of 850. At first glance, it looks like one of those classic “indexing starts at zero” bugs. But then I did, what you usually do when filling dictionaries: You check if the entry you want to add already exists. And it does.

And while this wouldn’t lead to a valid equation…

11095: 8 5 9 3 231

This line certainly does

11095: 8 1 2 7 95 8 2 7 2 2 99

And so I found the missing value for the final result…

Solution to the Second Part

Well, the second part took me a while because I was somehow trying to build this concatenation part into the current algorithm. We now have three operators, *, +, and || — but the last one should connect two integers.

Eventually, I had my heureka moment: Concatenation could be implemented as a mathematical operation, too.

If the left number is 123 followed by 45, we need to multiply 123 with 10^2 and then just add 45. Easy math.

Side note: I was cheating a little. I now know that 1095 is a duplicated entry, and I know it would lead to a valid result, so I am adding it hard-coded to get the final result.

Whats up, C++?

Rating: 6/12 – painful

Well, C++ is undoubtedly a powerful language, but for a reason it’s not beating all those scripting languages out there. Simple things are complicated because you have to implement them on your own. 5 out of 12 seems to be unfair for such a grown-up programming language, but this is simply my point of view. For me, right now, there’s no reason to use C++. Which absolutely does not mean that this language is not important! Nevertheless, using it felt like an adult pro developer. ;)

This task looks tricky, so I am once again choosing a simple language that I am familiar with: JavaScript ( which, by the way, was developed in just 10 days ).

With no further ado…

Solution to the First Part

There’s not much to explain. As expected, I just need to move the guard through the maze by increasing and decreasing rows and columns. When I hit an obstacle, I change direction. When I hit the border of the maze, I’ve solved the task. Easy peasy…

Solution to the Second Part

The second part is kind of tricky. They want us to find positions of additional obstacles that make the maze unsolvable.

Well, there’s an easy pattern behind it. You will see it if you visualize it:

An infinite loop occurs when you have four obstacles on each edge of a rectangle, like this:

So, all we need to do at every step on our path is to try to draw a rectangle. If it has three existing obstacles, we get one potential solution for an “artificial obstacle.

In case you’re wondering: I didn’t finish the second part. I know the solution approach, but seeing that I am three days behind already, I stopped working on it.

Whats up, JavaScript?

Rating: 9/12 – god ol’ frien’

Fun Fact: I was writing an algorithm to find prime numbers and found out that JavaScript is way faster than Python.

However, I really like JavaScript and often use it when I want to build small proof of concepts. It’s quick, runs everywhere (like in every browser), and is quite straightforward.

Despite the second bump in the journey to endless glory, I hope to see you again tomorrow!

It’s the fifth day of our outrageous adventure, and we are dealing with printer updates. To solve this riddle, we are getting back to the roots and our good ol’ friend Bash!

The good news here is that I don’t need to set up any particular environment because, thankfully, Bash is available on most systems out of the box.

Solution to the First Part

The solution is quite straightforward: We need to read the rules and the updates into an array, loop through all updates, and check if they somehow break the rules. That’s nothing too complicated. The only thing that I am probably doing unconventionally is how I check.

I am using a regular expression that is the opposite of the rule. If it matches, I know the update is invalid:

161.*?(,29)(,|$)

In this example, 29 is the actual rule name, and 61 should come after 29. With .*?, I am ungreedy checking for any characters in between, and the closing group just makes sure that after 29, there’s either another number or the end of the update.

Simple thing.

Solution to the Second Part

The second part looks way easier. They just want us to switch the position of the pages that we identified as erroneous.

Basically, it’s just something like this:

1updates[page_index]=$subsequent_page
2updates[subsequent_page_index]=$page

But… yeah, that’s the first bump in my journey. My code isn’t resulting in the correct answer, and it seems like I am missing a detail. To avoid wasting any more time, I’ll proceed to the next day… sorry folks :(

Whats up, Bash?

Rating: 8/12 – weird operators and syntax

The boss of all scripting languages: Bash. I still don’t understand most of the syntax rules, but thanks to AI, it’s not too hard to get things done in Bash. When working as a sysadmin, I loved automating processes and even building small user interfaces to make life on the command line easier. However, I guess there are better ways for that.

See you next day!

I Made It to Day 4!

And today’s task perfectly fits Excel. Some—if not most—people don’t like Excel and would never consider it a serious tool for data analytics, exploration, or even “engineering.” Well, you’re wrong about Excel. It’s quite powerful if you know how to use it. Let me show you…

The first part of the task is a kind of word riddle where we have to count occurrences of “XMAS” in a huge grid of letters. And actually, that’s pretty easy with Excel. I’d even say it’s easier than using a fully-fledged scripting tool.

Solution to the First Part

The input comes as a couple of lines. To make each character addressable, we first need to split each character into its own column. While I could use “Text to Columns,” the goal is to make this dynamic, so I will utilize a formula:

_LENGTH refers to [C2]

1[D1]=MID(B4;SEQUENCE(1;_LENGTH);1)

If you pull this formula down, you get a matrix like this (referring to the example input):

The following image shows all possible solutions:

We start with the most simple solution, where XMAS is plotted from the center to the right. The surrounding if-condition is temporary, only to make the result visible:

1=IF(CONCAT(G21;H21;I21;J21)="XMAS";"+";"")

It’s quite easy, right? We copy the formula (not the result of the cell!) to a text editor and modify it to get the other seven directions.

Eventually, you end up with eight lines looking like this:

1=IF(CONCAT(G21;H21;I21;J21)="XMAS";"+";"")
2=IF(CONCAT(G21;H22;I23;J24)="XMAS";"+";"")
3=IF(CONCAT(G21;G22;G23;G24)="XMAS";"+";"")
4=IF(CONCAT(G21;F22;E23;D24)="XMAS";"+";"")
5=IF(CONCAT(G21;F22;E23;D24)="XMAS";"+";"")
6=IF(CONCAT(G21;F20;E19;D18)="XMAS";"+";"")
7=IF(CONCAT(G21;G20;G19;G18)="XMAS";"+";"")
8=IF(CONCAT(G21;H20;I19;J18)="XMAS";"+";"")

To count the occurrences of all existing solutions, we could either build some kind of LAMBDA function or use a simple trick: We just cast the TRUE or FALSE result to an integer:

1=--(VERKETTEN(G21;H21;I21;J21)="XMAS")+
2    --(VERKETTEN(G21;H22;I23;J24)="XMAS")+
3    --(VERKETTEN(G21;G22;G23;G24)="XMAS")+
4    --(VERKETTEN(G21;F21;E21;D21)="XMAS")+
5    --(VERKETTEN(G21;F22;E23;D24)="XMAS")+
6    --(VERKETTEN(G21;F20;E19;D18)="XMAS")+
7    --(VERKETTEN(G21;G20;G19;G18)="XMAS")+
8    --(VERKETTEN(G21;H20;I19;J18)="XMAS")

And now apply it to our testing dataset. Copy the cell (not the formula!) next to our matrix. It should refer to the first column in the first row. Then drag it to the right and down to cover the whole area. You should end up with something like this:

The sum of the matrix is 18 — which is exactly the number of XMAS hidden in the test dataset. Now take the actual puzzle input, copy it to your sheet, and apply your magic formula to it. If you want to avoid dragging the formula to the right and down, you could use SEQUENCE again, but we’ll save that for later…

The good thing about the Excel approach is that the second part doesn’t make it more complicated. We stick with our strategy and now just look for all possible variations of X-MAS. There are four:

And our formula changes a little. We know that A has to be in the middle. We concatenate the surrounding four cells and just test for all four possible combinations of S and M, which leads us to something like this:

1=--(OR(CONCAT(F210;G209;G211;E211;E209)="AMSSM";
2    CONCAT(F210;G209;G211;E211;E209)="ASSMM";
3    CONCAT(F210;G209;G211;E211;E209)="ASMMS";
4    CONCAT(F210;G209;G211;E211;E209)="AMMSS"))

It’s even simpler than the previous one! Again, just make it reference your dataset, and you get the correct result!

Whats up, Excel?

Rating: 10/12 – understimated and powerful

I’m not gonna lie: I really love Excel. Since a couple of years, Excel also supports LAMBDA, which allows for more complex calculations and even functional programming. [You can even build an animated ticker in Excel]((shameless self-promotion)!

That said… see you next day!

I had initially planned to solve today’s riddle in BASIC—the first programming language I learned 30 years ago. However, it turns out that deploying a BASIC compiler into a container isn

Today, we need to parse a text to find all occurrences of mul(n, m), calculate the product of n and m, and finally compute the sum of all products. Let’s try to find a neat one-liner for that.

Solution of the first part

Here it is:

1with open("input.txt", "r") as file: print(sum([reduce(mul, ast.literal_eval(match.replace("mul", ""))) for match in re.findall("mul\(\d{1,3},\d{1,3}\)", file.read())]))

What am I doing? Well, let’s read it from the inside out:

First, we get all occurrences of mul(n, m), which is quite easy using regex:

1re.findall("mul\(\d{1,3},\d{1,3}\)

Then, we go through all matching groups using Python’s list comprehension feature:

1[match for match in re.findall("mul\(\d{1,3},\d{1,3}\)", file.read())]

But we’re not just using the plain result, which is a string like mul(4, 2). First, we remove mul to get the parentheses, then we evaluate the string into an actual tuple, like (4, 2):

1ast.literal_eval(match.replace("mul", ""))

Now that we have a list of integers, we can apply multiplication to each item:

1reduce(mul, (4, 2))

Each of the products will be fed into the final list. The sum() method simply adds up all the products in the list. The surrounding with ... as file: simply opens the file to read the content.

Solution of the second part

Well, the second part is a little trickier with three additional rules:

• Only sum up products of mul() if there’s the preceding keyword do().
• Ignore all mul() instructions after the keyword dont().
• All mul() instructions at the beginning, where there’s neither a dont() nor a do(), need to be considered, too.

This sounds easy at first glance, so I tried to figure out a way to simply adjust my regular expression. With no success, but a ton of frustration. However, I figured out a different way that is way more fun and demonstrates the obscure power of list comprehension:

1sum([sum([sum([reduce(mul, ast.literal_eval(match.replace("mul", ""))) for match in re.findall("mul\(\d{1,3},\d{1,3}\)", dos)]) for _j, dos in enumerate(donts.split("do()")[1:])]) for _i, donts in enumerate(file_contents.split("don't()")) if len(donts.split("do()")) > 1])

I guess you have questions? Fair enough. There you go, this is the expanded version:

1sum_of_all = 0
2for _i, donts in enumerate(('do()' + file_contents).split("don't()")):
3    if len(donts.split("do()")) > 1:
4        for _j, dos in enumerate(donts.split("do()")[1:]):
5            for match in re.findall("mul\(\d{1,3},\d{1,3}\)", dos):
6                print(f'string: {match} - list: {ast.literal_eval(match.replace("mul", ""))} -  product: {reduce(mul, ast.literal_eval(match.replace("mul", "")))}')
7                sum_of_all += reduce(mul, ast.literal_eval(match.replace("mul", "")))
8
9print(f'sum: {sum_of_all}')

The idea is that I simply split the whole text by the string dont() in the first for loop. This returns a list of items, where I know that every mul() must be ignored until I encounter a do() in the string.

This leads to the next operation: Instead of parsing the string, I split each of the strings by the keyword do(). If this string contains do(), I get a list of 2 items. If not, I can ignore the whole string in my outer loop.

Otherwise, I take all items except the first one, and now I can apply the logic from above: using the regular expression to get the list of integers, then multiplying and summing up.

There’s a small pitfall: The third rule says that all instructions at the beginning of the string, until the first do() or dont(), need to be considered. The simplest way to handle this is by just adding do() to the initial input string.

In case you’re wondering how to go from the seemingly simple connection of nested loops to this nasty nested list comprehension—you achieve it step by step:

First, we iterate through the raw file content:

1[donts for _i, donts in enumerate(('do()' + file_contents).split("don't()"))]

Then, we replace dont() with the next loop:

1[[dos for _j, dos in enumerate(donts.split("do()")[1:])] for _i, donts in enumerate(('do()' + file_contents).split("don't()"))]

Don’t forget to add a condition to skip results with a length of 1:

1[[dos for _j, dos in enumerate(donts.split("do()")[1:])] for _i, donts in enumerate(('do()' + file_contents).split("don't()")) if len(donts.split("do()")) > 1]

Then, we can replace do() with the following loop:

1[[[match for match in re.findall("mul\(\d{1,3},\d{1,3}\)", dos)] for _j, dos in enumerate(donts.split("do()")[1:])] for _i, donts in enumerate(('do()' + file_contents).split("don't()")) if len(donts.split("do()")) > 1]

Now we are at the point where we handle the matching groups, so we replace match with everything we need to get a product out of the stringified list of integers:

1[[[reduce(mul, ast.literal_eval(match.replace("mul", ""))) for match in re.findall("mul\(\d{1,3},\d{1,3}\)", dos)] for _j, dos in enumerate(donts.split("do()")[1:])] for _i, donts in enumerate(('do()' + file_contents).split("don't()")) if len(donts.split("do()")) > 1]

And eventually, we get the sum of each nested list:

1sum([sum([sum([reduce(mul, ast.literal_eval(match.replace("mul", ""))) for match in re.findall("mul\(\d{1,3},\d{1,3}\)", dos)]) for _j, dos in enumerate(donts.split("do()")[1:])]) for _i, donts in enumerate(('do()' + file_contents).split("don't()")) if len(donts.split("do()")) > 1])

Looks great, doesn’t it?

Whats up, Python?

Rating: 14/12 – nice and friendly, always again

Uncontroversial—one of Python’s best features is list comprehensions, which also work for dictionaries, sets, and tuples, by the way. List comprehensions have so many features; besides the way you can nest them, you can add conditions in two ways. Either when you add an item to the output list…

1[x if x % 2 == 0 else -x for x in range(10)]

…or before:

1[x for x in range(10) if x % 2 == 0]

You can loop through multiple lists (et al):

1[(x, y) for x in range(3) for y in range(2)]

You can not only nest them, but also serialize multiple loops to flatten a nested input:

1nested = [[1, 2], [3, 4], [5, 6]]
2[x for sublist in nested for x in sublist]

And there are probably a couple more that I don’t even know.

See you next day…

This task is about testing if integers in a list are increasing or decreasing. I’ll try Rust, and I assume I need nerves of steel for this exercise.

Setting Up the Development Environment

Rust comes in a container, too. Here’s the Dockerfile:

 1# Dockerfile
 2FROM rust:latest
 3
 4# Install additional tools you might need
 5RUN apt-get update && apt-get install -y \
 6    git \
 7    curl \
 8    wget \
 9    build-essential
10
11# Set the working directory
12WORKDIR /workspace
13
14# Copy project files (if any)
15COPY . .
16
17# Default command
18CMD ["bash"]

And here’s the devcontainer.json to develop inside that container made of steel:

 1{
 2    "name": "Rust Development Container",
 3    "build": {
 4        "dockerfile": "../Dockerfile"
 5    },
 6    "customizations": {
 7        "vscode": {
 8            "extensions": [
 9                "rust-lang.rust-analyzer",
10                "serayuzgur.crates",
11                "tamasfe.even-better-toml"
12            ]
13        }
14    },
15    "remoteUser": "root",
16    "features": {
17        "ghcr.io/devcontainers/features/rust:1": {}
18    }
19}

Rust also requires a Cargo.toml file that defines your project and dependencies:

1# Cargo.toml
2[package]
3name = "rust-docker-project"
4version = "0.1.0"
5edition = "2024"
6
7[dependencies]
8# none, ehehe

That’s all we need. Build it with docker build and connect VS Code. The source file is located inside the src folder. You can run your code like this

1cargo build
2cargo run

The algorithm itself isn’t that tricky. You’ll find everything on Github .

Whats up, Rust?

Rating: 4/12 – Wouldn’t use it freely

Don’t get me started. I didn’t know that Rust has so many strange rules. There’s this interesting concept of ownership: values belong to a variable as long as you don’t assign that variable to another variable. If you pass a value from one vector to another, you can’t access it in the original vector anymore.

And then there are many important operators scattered across the code, like:

• & for borrowing (referencing a value without taking ownership)
• * for de-referencing (accessing the value a reference points to)

What I do like, though, is the error handling. Rust uses Ok and Err to make handling errors much cleaner and explicit:

1Err("Value not valid")
2Ok(())

This approach forces you to handle potential issues early on, which is great for reliability.

See you next day…

As the challenge at AoC is not about the syntax but the algorithm, I decided to try a different approach this year: a new language every day. I am probably not going to explain the code in detail, but I will point out characteristics of each particular language that I think are worth mentioning. Let’s see how far I get. Please don’t take my judgement to serious.

Disclaimer: Of course, I am using AI to get the actual syntax, but not for the algorithm itself. I usually use ClaudeAI for that, as I’ve experienced the best results with the free version there.

On the first day, I’m starting with F# — the sister of C# — which focuses on functional programming.

Setting Up the Development Environment

First, I need to set up my development environment. Since I don’t want to pollute my system with tons of different interpreters, I prefer to work in Docker containers. Above all, this requires the Dev Containers extension . To make VS Code ready for F#, I also need the Ionide extension .

Here is the Dockerfile for a containerized F# environment:

 1# Use official .NET SDK image with F# support
 2FROM mcr.microsoft.com/dotnet/sdk:8.0
 3
 4# Install additional tools
 5RUN apt-get update && \
 6    apt-get install -y \
 7    git \
 8    curl \
 9    vim \
10    sudo
11
12# Create vscode user with sudo permissions
13RUN groupadd -r vscode \
14    && useradd -m -s /bin/bash -g vscode -G sudo vscode \
15    && echo "vscode ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/vscode \
16    && chmod 0440 /etc/sudoers.d/vscode
17
18# Set working directory and ensure proper ownership
19WORKDIR /app
20RUN chown -R vscode:vscode /app
21
22# Ensure .NET tools are in PATH
23ENV PATH="/home/vscode/.dotnet/tools:${PATH}"
24
25# Set the default user to vscode
26USER vscode
27
28# Ensure .NET tools are installed for the vscode user
29RUN dotnet tool list --global || true
30
31# Default command
32CMD ["/bin/bash"]

Developing inside a container requires /.devcontainer/devcontainer.json to look like this:

 1{
 2    "name": "F# Dev Environment",
 3    "build": {
 4        "dockerfile": "../Dockerfile",
 5        "context": ".."
 6    },
 7    "remoteUser": "vscode",
 8    "workspaceFolder": "/app",
 9    "customizations": {
10        "vscode": {
11            "extensions": [
12                "Ionide.ionide-fsharp",
13                "ms-dotnettools.csharp"
14            ]
15        }
16    },
17    "mounts": [
18        "source=${localWorkspaceFolder},target=/app,type=bind,consistency=cached"
19    ],
20    "containerUser": "vscode",
21    "remoteEnv": {
22        "DOTNET_CLI_HOME": "/app"
23    }
24}

Now, after building and running the container with docker-compose build; docker-compose up -d, I can connect to the container by either pressing CMD+SHIFT+P or using the small blue button on the bottom left, selecting the option “Connect to Dev Container.”

Day 1 - About differences and products

( task - solution )

The first day consists of two simple tasks, so I won’t go into too much detail:

1. Order two lists incrementally, then get the difference for each list item and sum all differences up.
2. For each item on the first list, count the occurrences on the second list, compute the product i * frequency(i), and then get the sum of all products.

Whats up, F#?

8 of 12, would like to stay in touch

Well, I certainly havent had the chance to get familar with F#, but at the first glance I like how you can connect output to adjecent commands using the pipe operator:

1let readListFromFile (filePath: string) =
2    File.ReadAllLines(filePath) 
3    |> Array.map int
4    |> Array.toList

This makes things quiete easy, resulting in a clearer and more concise coding style. Seems like F# is a good choice for hands one problem solving.

See you next day…

Follow my journey on how to build an algorithm that lets you create or solve Sudoku puzzles!

Follow my steps or jump ahead — this is the Jupyter notebook: https://gist.github.com/nickyreinert/966299333429ef685338e5feb5056126

Warmup — introduction

Before we start, we need to get familiar with how Sudoku works. I suggest we number each cell in the giant grid from 1 to 81. This way, we can programmatically loop from the top-left to the bottom-right to find valid solutions:

Sudoku has three rules: a number can’t exist twice in the same column, row, or section. So, first, we need a way to identify the column, row, and section for each position.

Every row contains 9 cells, so the row index is simply:

ceil(position / 9)

Or if you prefer double slash (//), which works like a “floor” function:

((position — 1) // 9) + 1

Columns probably also have something to do with 9 (because 9 is kind of a big deal in Sudoku). The first nine cells are exactly addressing their corresponding columns, obviously. So, in the upcoming rows (positions 10 and onward), we just need to reduce it back to 1 to 9. We can use modulo for that!

((position — 1) % 9) + 1

The section index is a little trickier. It’s a combo of the row and column index and some tweaks. There’s probably a clever mathematical approach, or you can just trial-and-error your way through (my usual strategy). Here’s a starting point:

We take the floor of a third of each position now. It seems like we could just sum them to get the fitting section index, with a little adjustment. Here’s the magic:

And that’s the formula to get the section index:

(col // 3) + (((row // 3) — 1) * 3)

First steps

Well, now that we know how to address each row, column, and section, let’s dive into the actual algorithm.

Solution approach

The basic idea is to iterate through the positions up to 81. At every step, we choose a number from 1 to 9 and check if it already exists in the corresponding row, column, or section (we’ll call these containers). If it does, the number is blocked for the current position, and we have to check the next number. If we can’t find a valid number for the current position, we need to roll back to the previous one and choose a different number. We repeat this process until we either reach position 81 — meaning we’ve found a valid Sudoku — or have to backtrack more.

Implementation in Python

We start by initializing four dictionaries to keep track of the numbers in each container. The solutions dictionary holds (surprise!) the valid numbers for the entire grid.

def reset_containers():  
    global rows, cols, sections, solutions  
    rows = {i: [] for i in range(1, 10)}  
    cols = {i: [] for i in range(1, 10)}  
    sections = {i: [] for i in range(1, 10)}  
    solutions = {}  

reset_containers()  

print(sections, '\n', rows, '\n', cols)

Result:

{1: [], 2: [], 3: [], 4: [], 5: [], 6: [], 7: [], 8: [], 9: []}   
{1: [], 2: [], 3: [], 4: [], 5: [], 6: [], 7: [], 8: [], 9: []}   
{1: [], 2: [], 3: [], 4: [], 5: [], 6: [], 7: [], 8: [], 9: []}

The visualize function takes the solution dictionary and returns a nice grid, displaying the numbers — you’ll find it in the notebook I referenced at the top!

def visualize(solution):  
    # see Jupyter notebook referenced on the top

We’ve already figured out how to get the actual row, column, or section from a given position. Now, let’s translate into Python and check if we’re on the right track:

print('pos', '\t', 'row', '\t', 'col', '\t', 'section')  
for position in range(1, 82):  
    idx_row = math.ceil(position / 9)  
    idx_col = ((position - 1) % 9) + 1  
    idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  
    print(position, '\t', idx_row, '\t', idx_col, '\t', idx_section)

Output:

pos   row   col   section  
1   1   1   1  
2   1   2   1  
3   1   3   1  
4   1   4   2  
5   1   5   2  
6   1   6   2  
7   1   7   3  
8   1   8   3  
9   1   9   3  
10   2   1   1  
…

Yes, we are! Every position is correctly mapped to the correct container!

Solution checking and iterating

The first version is quite simple: We iterate through the positions and add a number that does not exist in one of the corresponding containers.

reset_containers()  

number = 1  

for position in range(1, 82):  
    idx_row = math.ceil(position / 9)  
    idx_col = ((position - 1) % 9) + 1  
    idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  
    if (number not in rows[idx_row] and number not in cols[idx_col] and number not in sections[idx_section]):  
        solutions[position] = number  
        rows[idx_row].append(number)  
        cols[idx_col].append(number)  
        sections[idx_section].append(number)  
        number += 1  

visualize(solutions)

Seems like our basic loop is working fine. Let’s add some fine-tuning and limit it to the numbers from 1 to 9. We can just introduce an inner loop to keep things simple:

Output:

Seems like our basic loop is working fine. Let’s add some fine-tuning and limit it to the numbers from 1 to 9. We can just introduce an inner loop to keep things simple:

reset_containers()  

number = 1  

for position in range(1, 82):  
    idx_row = math.ceil(position / 9)  
    idx_col = ((position - 1) % 9) + 1  
    idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  
    for number in range(1, 10):  
        if (number not in rows[idx_row] and number not in cols[idx_col] and number not in sections[idx_section]):  
            solutions[position] = number  
            rows[idx_row].append(number)  
            cols[idx_col].append(number)  
            sections[idx_section].append(number)  
            break  

visualize(solutions)

Output:

There are a lot of empty spots in the grid, what are we missing? Let’s introduce a conditional that stops when there’s no solution:

reset_containers()  

number = 1  

for position in range(1, 82):  
    idx_row = math.ceil(position / 9)  
    idx_col = ((position - 1) % 9) + 1  
    idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  
    for number in range(1, 10):  
        if (number not in rows[idx_row] and number not in cols[idx_col] and number not in sections[idx_section]):  
            solutions[position] = number  
            rows[idx_row].append(number)  
            cols[idx_col].append(number)  
            sections[idx_section].append(number)  
            break  

    if not position in solutions:  
        break  

visualize(solutions)

Output:

Quite good, but just stopping the loop when we hit a dead end probably won’t get us to a solution. We need to keep track of dead ends, and we need the ability to go back, choose another path, and try again. Let’s start with a container that holds “impossible” paths for each of our 81 positions.

We’ll also switch from a limited for-loop to an “infinite” while-loop because now we’re not just incrementing from 1 to 81 — we’ll be going back and forth. Of course, it’s always a good idea to use a stop condition, like max_iterations, so we don’t accidentally run into an actual endless loop when using while.

Let’s put all of that into a new “big” init function!

def reset_containers():  
    global rows, cols, sections, solutions, dead_ends  
    rows = {i: [] for i in range(1, 10)}  
    cols = {i: [] for i in range(1, 10)}  
    sections = {i: [] for i in range(1, 10)}  
    solutions = {}  
    dead_ends = {i: [] for i in range(1, 82)}

Now, if the current iteration doesn’t result in a solution, we don’t just stop processing. We go back one step:

position -= 1

And we archive the solution from the previous position as a “dead end solution”:

dead_ends[position].append(solutions[position])

Then we have to update our containers, as the solution is no longer valid:

rows[idx_row].remove(solutions[position])  
    cols[idx_col].remove(solutions[position])  
    sections[idx_section].remove(solutions[position])  
    del solutions[position]

Let’s see how this performs!

reset_containers()

print('iter', '\t', 'pos', '\t', 'row', '\t', 'col', '\t', 'section', '\t\t', 'number')  

iteration = 1  
position = 1  
max_iterations = 10_000  
stop = False  

while True:  
    idx_row = math.ceil(position / 9)  
    idx_col = ((position - 1) % 9) + 1  
    idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  
    for number in range(1, 10):  
        print(iteration, '\t', position, '\t', idx_row, '\t', idx_col, '\t', idx_section, '\t\t', number)  

        iteration += 1  
        if (iteration > max_iterations): stop = True; break  

        if (number not in rows[idx_row] and \  
            number not in cols[idx_col] and \  
            number not in sections[idx_section] and \  
            number not in dead_ends[position]):  
            solutions[position] = number  
            rows[idx_row].append(number)  
            cols[idx_col].append(number)  
            sections[idx_section].append(number)  
            break  

    if stop: break  

    if not position in solutions:  

        position -= 1  
        if (position <= 0): stop = True; break  

        dead_ends[position].append(solutions[position])  

        print(f'rolling back {solutions[position]}')  

        idx_row = math.ceil(position / 9)  
        idx_col = ((position - 1) % 9) + 1  
        idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  

        rows[idx_row].remove(solutions[position])  
        cols[idx_col].remove(solutions[position])  
        sections[idx_section].remove(solutions[position])  
        del solutions[position]  
    
    else:  
        position += 1  

    if (position > 81): break  

print(f'Done after {iteration - 1:,.0f} iterations.')  

visualize(solutions)

Output:

An empty grid? Why for flux sake? Well, our housekeeping is incomplete — we didn’t properly clean up the dead end container:

dead_ends[position] = []

Let’s take this chance to adjust our container reset function. It will now reset all containers or just the ones for a given position only:

def reset_containers(position = None):  
    global rows, cols, sections, solutions, dead_ends  

    if position is None:  
        rows = {i: [] for i in range(1, 10)}  
        cols = {i: [] for i in range(1, 10)}  
        sections = {i: [] for i in range(1, 10)}  
        solutions = {}  
        dead_ends = {i: [] for i in range(1, 82)}  
    else:  
        idx_row = math.ceil(position / 9)  
        idx_col = ((position - 1) % 9) + 1  
        idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  

        dead_ends[position].append(solutions[position])  
        rows[idx_row].remove(solutions[position])  
        cols[idx_col].remove(solutions[position])  
        sections[idx_section].remove(solutions[position])  
        del solutions[position]

And now the main loop:

def solve_game(max_iterations = 10_000):  

    global rows, cols, sections, solutions, dead_ends  
    
    reset_containers()  

    iteration = 1  
    position = 1  
    stop = False  

    # print('iter', '\t', 'pos', '\t', 'row', '\t', 'col', '\t', 'section', '\t\t', 'number')  

    while True:  
        idx_row = math.ceil(position / 9)  
        idx_col = ((position - 1) % 9) + 1  
        idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  
        for number in range(1, 10):  
            # print(iteration, '\t', position, '\t', idx_row, '\t', idx_col, '\t', idx_section, '\t\t', number)  

            iteration += 1  
            if (iteration > max_iterations): stop = True; break  

            if (number not in rows[idx_row] and \  
                number not in cols[idx_col] and \  
                number not in sections[idx_section] and \  
                number not in dead_ends[position]):  
                solutions[position] = number  
                rows[idx_row].append(number)  
                cols[idx_col].append(number)  
                sections[idx_section].append(number)  
                break  

        if stop: break  

        if not position in solutions:  

            dead_ends[position] = [] # if there were any dead ends on this position, remove them now, because we roll back  

            position -= 1  
            if (position <= 0): break  

            reset_containers(position) # and clean up containers for the previous position  

        else:  
            position += 1  

        if (position > 81): break  

    print(f'Done after {iteration - 1:,.0f} iterations.')  

solve_game(max_iterations=10_000)  

visualize(solutions)

Output:

Wohaa! Looks like our little loop is already producing a valid result. But how does it perform when we add some numbers to the solution?

To work with a pre-filled grid, we need to implement another container — let’s call it target. We’ll add a simple check: If the target contains the current position, we’ll treat it as a constant. Copy it over to the other containers and skip to the next position:

if position in target:   
        number = grid[position]  
        rows[idx_row].append(number)  
        cols[idx_col].append(number)  
        sections[idx_section].append(number)  
        position += 1  
        continue

We also need to adapt the part where we move back one step when no solution can be found. If we move back from a regular position to a pre-filled one, we should instantly jump back to the last regular position. Otherwise, we’d get stuck in a loop. So, let’s declare last_regular_position, which points to the last regular cell before a pre-filled one, and handle the backtracking that way.

This should help avoid unnecessary loops and keep things running smoothly.

Our loop is growing:

def solve_game(debug = False, max_iterations = 10_000, target = {}):  

    global rows, cols, sections, solutions, dead_ends  

    reset_containers()  

    iteration = 1  
    position = 1  
    stop = False  
    start_of_occupied_area = 82  

    if debug: print('iter', '\t', 'pos', '\t', 'row', '\t', 'col', '\t', 'section', '\t\t', 'number')  

    # mandatroy: register all target position with our containers  
    for target_position in target:  
        idx_row = math.ceil(target_position / 9)  
        idx_col = ((target_position - 1) % 9) + 1  
        idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  
        rows[idx_row].append(target[target_position])  
        cols[idx_col].append(target[target_position])  
        sections[idx_section].append(target[target_position])  
        
    while True:  
        idx_row = math.ceil(position / 9)  
        idx_col = ((position - 1) % 9) + 1  
        idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  

        if (position in target):   
            number = target[position]  
            solutions[position] = number  
            rows[idx_row].append(number)  
            cols[idx_col].append(number)  
            sections[idx_section].append(number)  

            if position < start_of_occupied_area: start_of_occupied_area = position  

            position +=1  
            continue  

        start_at = 1  
        if len(dead_ends[position]) > 0:  
            start_at = max(dead_ends[position]) + 1  

        for number in range(start_at, 10):  
            if debug: print(iteration, '\t', position, '\t', idx_row, '\t', idx_col, '\t', idx_section, '\t\t', number)  

            iteration += 1  
            if (iteration > max_iterations): stop = True; break  

            if (number not in rows[idx_row] and \  
                number not in cols[idx_col] and \  
                number not in sections[idx_section] and \  
                number not in dead_ends[position]):  
                solutions[position] = number  
                rows[idx_row].append(number)  
                cols[idx_col].append(number)  
                sections[idx_section].append(number)  

                break  

        if stop: break  

        if not position in solutions:  
            
            dead_ends[position] = []  
            
            if (position - 1) in target:  

                position = start_of_occupied_area - 1  

            else:  
                
                position -= 1  

            reset_containers(position)   

        else:  

            position += 1  

        if (position > 81): break  

    print(f'Done after {iteration - 1:,.0f} iterations.')  

solve_game(debug = False, max_iterations = 5_000, target = {16: 2})  

visualize(solutions)

Output:

Farn-tastic! But don’t get too hyped, we’re not there yet. What about different solutions, variations? When we reach the last cell at position 81, in row 9 and column 9, we find exactly one solution for each previous cell. But how do we check for other options? To understand our next approach, imagine the solution as a huge 81-digit number, with all possible solutions branching out like a massive tree. The leftmost branch is the smallest number, and we’re going to find “higher” numbers, which are also valid solutions.

Let’s start with a simple idea: What if, as soon as we reach the end of the grid, we:

• Go back to the first item
• Store all numbers from the current solution to ignore them in the next iteration
• Switch to a new, empty solutions container
• And start over from the first position?

Well, I’ll spare us the effort of implementing it and instead ask you to think it over for yourself. You’ve got 5 milliseconds until the next paragraph starts…

Ka-ching! (Do you remember [Lockerz from 2009](Why did I invest my time into that scam instead of Bitcoin? Whatever…)

It wouldn’t work! The first cell accepts 1 to 9 as possible options — our main branches of a tree. At the second level, which is the second cell, again we have 1 to 9 as possible values. Solution 1 already gave us this path:

1 - 2 - …

But if we ruled out the 1 when iterating through the second solution, we’d also rule out another possible path:

1 - 3 - …

So, instead of resetting to the first cell, the top of the tree, we just move back to the last cell — or the last node in the path or branch, however you want to call it. Right now, it looks like this:

… - 6 - 4 - 2

Think of the solution as a big number:

…642

From the previous run, we know that the following paths (or numbers) aren’t possible:

…611  
…621  
…631  
…641  
…642 - ka-ching!

Because, for some reason, the loop always took us back until we eventually hit the 4 and the 2. Now, what happens if we search for the next solution by going back to 81 again, but this time adding the 2 to our list of “dead ends”? The loop will run as it has before:

…643 - not possible, step back  
…65 - not possible, step back  
…7 - not possible, step back

It will keep doing that until — spoiler alert — it reaches position 66 (row 8, col 3):

…642

And it will try again, testing all possible numbers until it eventually hits:

…648

Which is the start of a new possible solution!

Sounds about right, doesn’t it?

Let’s implement it! First, a small adjustment to our init methods and our solutions container, which now holds multiple solutions:

solutions[solution_index][position]

Of course, we also need to adjust the function’s signature and how we reset our solution container. The way we address positions in the solution deck changes, too. It’s a list now, which makes it easier to slice the first n items of it (honestly, I don’t know why we didn’t do that from the beginning — it’s not really harder to handle. Guess that’s what they call “learning by doing” ;) ).

def reset_containers(solution_index, position = None):  
    global rows, cols, sections, solutions, dead_ends, burned_numbers  


    if position is None and solution_index == 1:  
        rows = {i: [] for i in range(1, 10)}  
        cols = {i: [] for i in range(1, 10)}  
        sections = {i: [] for i in range(1, 10)}  
        solutions[solution_index] = {}  
        dead_ends = {i: [] for i in range(1, 82)}  
    else:  
        idx_row = math.ceil(position / 9)  
        idx_col = ((position - 1) % 9) + 1  
        idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  

        if position in solutions[solution_index]:  
            number = solutions[solution_index][position]  
            del solutions[solution_index][position]  
        elif position in burned_numbers:  
            number = burned_numbers[position]  
            
        dead_ends[position].append(number)  
        rows[idx_row].remove(number)  
        cols[idx_col].remove(number)  
        sections[idx_section].remove(number)

Our solution method changes a little. Instead of stopping the loop when position > 81, we add the current solution to our “dead end” container and then go back to position 81.

def solve_game(debug = False, max_iterations = 10_000, target = {}):  

    global rows, cols, sections, solutions, dead_ends, burned_numbers  

    solution_index = 1  
    solutions = {solution_index: {}}  
    reset_containers(solution_index)  

    report_success = False  

    iteration = 1  
    position = 1  
    stop = False  
    burned_numbers = {} # keeps the previous solution  

    start_of_occupied_area = 82  

    if debug: print('solution', '\t', 'iter', '\t', 'pos', '\t', 'row', '\t', 'col', '\t', 'section', '\t\t', 'number')  

    for target_position in target:  
        idx_row = math.ceil(target_position / 9)  
        idx_col = ((target_position - 1) % 9) + 1  
        idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  
        rows[idx_row].append(target[target_position])  
        cols[idx_col].append(target[target_position])  
        sections[idx_section].append(target[target_position])  
        
    while True:  

        if position <= 81:  

            idx_row = math.ceil(position / 9)  
            idx_col = ((position - 1) % 9) + 1  
            idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  

            if (position in target):   
                solutions[solution_index][position] = target[position]  

                if position > 1 and position < start_of_occupied_area: start_of_occupied_area = position  

                position +=1;   
                continue  

            # if the current position has a burned number from the previous solution  
            # we can take this + 1 as the next starting point  
            # this should only happens once for each run when move backwards  
            start_at = 1  
            if position in burned_numbers:  
                last_found_number = burned_numbers[position]  
                start_at = last_found_number + 1  
                # note how we clean or contains only, not the actual burned_numbers record  
                # because if this iteration is not succesful, we need a reference!  
                del burned_numbers[position]  

            elif len(dead_ends[position]) > 0:  
                start_at = max(dead_ends[position]) + 1  

            for number in range(start_at, 10):  
                if debug: print(solution_index, '\t\t', iteration, '\t', position, '\t', idx_row, '\t', idx_col, '\t', idx_section, '\t\t', number)  

                iteration += 1  
                if iteration % (max_iterations // 10) == 0: report_success = True  
                if (iteration > max_iterations): stop = True; break  

                if (number not in rows[idx_row] and \  
                    number not in cols[idx_col] and \  
                    number not in sections[idx_section] and \  
                    number not in dead_ends[position]):  

                    solutions[solution_index][position] = number  
                    
                    rows[idx_row].append(number)  
                    cols[idx_col].append(number)  
                    sections[idx_section].append(number)  
                    break  

            if stop: break  
        
            if not position in solutions[solution_index]:  
        
                dead_ends[position] = []  

                # this is also very important: when we are at this point  
                # we couldn't find a solution for the current position  
                # so we step back to the next "free" cell  
                # that is not blocked by a "target"  

                if (position - 1) in target:  

                    position = start_of_occupied_area - 1  

                else:  
                    
                    position -= 1  

                # and then we clear the containers for this position, to have a fresh start  
                reset_containers(solution_index, position)  

            else:  

                position += 1  

        else:             

            idx_row = math.ceil((position - 1) / 9)  
            idx_col = ((position - 2) % 9) + 1  
            idx_section = math.ceil(idx_col/3) + ((math.ceil(idx_row/3) - 1) * 3)  

            # we found a complete solution, now record the numbers as "burned" for the next run  
            # for the next solution  
            
            if report_success == True: print(f'{solution_index} solutions completed after {iteration - 1:,.0f} iterations.'); report_success = False  

            # take over previous solutions as "burned numbers"  
            dead_ends = {i: [] for i in range(1, 82)}  

            # and also some house keeping, as   
            # we crawl back when trying additional solutions  
            # we only fill the part of the tree  
            # that actually changes, that's why we need to take over  
            # everything that did not change:  
            if solution_index >= 2:  
                solutions[solution_index] = {pos: solutions[solution_index][pos] if pos in solutions[solution_index] else solutions[solution_index - 1].get(pos) for pos in solutions[solution_index - 1]}  
                
            burned_numbers = solutions[solution_index].copy()  

            # it took me some time to find that out:   
            # basically we are telling the script to rewind   
            # as we do when we don't find a solution  
            # so we also need to remove the current number from our  
            # containers  
            number = solutions[solution_index][81]  
            rows[idx_row].remove(number)  
            cols[idx_col].remove(number)  
            sections[idx_section].remove(number)  

            position = 81  
            solution_index += 1  
            solutions[solution_index] = {}  

    print(f'Done after {iteration - 1:,.0f} iterationsm found at least {solution_index - 1} solutions')  

solve_game(debug=False, max_iterations=1_000_000, target = {80: 7})  

visualize(solutions[len(solutions) - 1])

And that’s our achievment:

Our brute-force Sudoku solving algorithm is ready! Well, I bet there are a couple of steps to optimize — not only code-style-wise, but also to make it run faster. I’d say that’s your homework. There are 6,670,903,752,021,072,936,960 possible paths (that’s 6 sextillion and a bit more, it can be writte as ~6 x 10^21, which is a reference to Blade Runner: Dr. Ana Stelline was born on the 10/6/21). Can you catch them all? How long does it take on your machine to get 1 million solutions?

Just one more thing… I asked ChatGPT to build a small function that checks if all solutions are valid. Wait, what? ChatGPT? Why haven’t we asked it earlier?

Because developing algorithms is fun?

You got it! Whatever, here we go:

def check_sudoku(solutions):  

    print('All good!')

Just kidding — you’ll find the actual implementation in the notebook. Before I send you off into your well-deserved time off, let’s quickly go over some visualizations. I “discovered” 295,295 solutions (took me 100 million iterations and roughly 45 seconds), and this is how they look like in an image:

Finding 300,000 solutions out of 6 sextillion possibilities is like picking a single grain of sand from all the sand on Earth.

For the following visualization, I calculated the difference between each solution — remember, they’re all essentially enormous numbers!

There are some interesting points to observe. On the symlog scale of outliers, you can clearly see bands of numbers. There’s also one outlier way out of the region, meaning that from solution 149,652 to 149,653, the loop has to revert to a very high position in our “solution tree.”

Can you find any more salient features? Let me know!

In the first part we take care of the preparations, in the second part it goes to the actual algorithm. The source code is on Github .

What will you learn?

In addition to creating a SHA-256, you will be familiar with the handling of binary numbers and binary arithmetic operations such as XOR, AND, etc. However, I assume that there is a basic understanding of binary numbers, the focus is on the algorithm. 10 should stand in your head either for the ten or one two. (Or also twelve, if you like the duo decimal system.)

Foreword

If you lower this to a maximum amateur description, the following happens with a crytological hash function: An output text of any length is processed in such a way that a result text (the hash) with the same length is always created. It is almost impossible to calculate the output text from the hash. In addition, you can almost certainly assume that each output text creates a different hash. If I change only one character, it affects the original text drastically. Such an algorithm is therefore predestined to be attached to texts, i.e. messages. This is why we also speak of a checksum.

And that is the foundation of a technology that is making more and more talks in the recent past: the blockchain, the basis for cryptocurrencies such as Bitcoin. With the blockchain, and even that only amateurly broken down, the entries of the “cash book” are securely against manipulation, because the change of a historical value (e.g. booking process) would inevitably result in a drastic change in the checksums generated from it. To take the blockchain apologist out of the wind out of the sails, I quote Fefe, meaning: It is also easier . I use Bitcoin here only as a buzzword, for marketing reasons. :]

But to motivate you to read on, an important note:

The algorithm is used to calculate the next records of the blockchain. Strictly speaking, a certain hash is setting, which is to be calculated (the notorious mining). The reward for the correct calculation is Bitcoins. The problem: This calculation is very, very complex, because as already written above: it only works in one direction. The miners therefore have to make unspeakable many calculations in order to calculate a target value. And the miner who carries out the calculation the fastest is also rewarded for it. So if you’re looking forward to optimizing the algorithm, you can get out big in the mining business. That sounds like a challenge, doesn’t it? ;)

Source: https://peakd.com/german/-marcus0alameda/dagobert-gold-bitcoin-perfect assembly

Disclaimer: I have recreated the whole algorithm in Python. Python is certainly not the best option to calculate SHA-256 from performance view and dealing with binary or hexadecimal values is somewhat uncomfortable. Thanks to Jupyter, Python is most likely to describe a complex algorithm step-by-step.

Introduction

Before we get involved with the loops, we need to take care of a few functions that we will use later to mix through binary numbers a little.

In the following, I renounce the prefixes of the number system, such as 0b binary, to keep the text clear. I assume that the following is known: 0 -> False and 1

Note 2: In the context of SHA-256, one word (dual word) corresponds to exactly 32 bits. In general, 1 Word corresponds to 2 bytes of 16 bits.

The Explicit Or (XOR)

The explicit mean or (either-or) is an elementary logical, bitwise operator. The outcome of the operation is only true if exactly a state is true (in comparison, the result of the “simple” OR is true, by the way, if at least one operand is true or both).

So two values are processed as follows:

XOR: only if exactly one value is true (1), the corresponding position is true in the result (1)

The implementation in Python is done with the circumflex :

1110 to 100
2010

The logical and (AND)

The AND operator is also quite common and comparatively simple. Analogous to XOR, the result is true when exactly both (or all) operands are true.

AND: Only if both values of a single area are true, the passage is true in the result

The implementation in Python is done with the commercial and :

1110 & 100
2100

The Negation (Nope?)

Now it becomes strange: There is also an operator for this: The bitwise operator negation revolves values around. 0 becomes 1, from 1 will be 0.

The negation reverses values bitwise. No more but no longer not less.

The implementation in Python is done with the tilde à my favourite character!

1110
2001

The Shift Operation

The shift function is an elementary binary computing operation in which the individual places of a binary value are moved to the left or right. The vacancies on the other side will be filled with 0.

Shift to the left around a place, from 6 to 12

And now there is hopefully a positive kink in the learning curve: If you look closely, you will not notice something and let me assure you, it’s not a coincidence: 12 is the product of 6 and 2. This indicates an interesting side effect: A shift comes to a multiplication or. Division with 2 equal. A shift around several digits corresponds to a multiplication with a potency to the base 2 exists. Sounds complicated, therefore an example:

Instead of 139 * 2, you can shift the binary representation of 139, so 10001011, to 17 digits to the left. The result: 1000101100000000000000000. Count after, right of 1 one there are now 17 zeros.

In Python, the binary shift with the double arrow is implemented:

1110 >> 1 
2011110 to 2
3€ 000

The Rotate function

Rotate means that one shifts the values of a (binary) number in one direction. And this is best explained by an example. The following number series should be rotated by one counter to the left. The number on the left side falls out and we turned on the right. The other numbers move one position to the left.

Rotate of a binary value by one place to the left, from 6will 5

This works in both directions and with any number of places. The corresponding function ([Kudos an action](looks like this):

1def rotate(value, rotations, width à 32):    
2if int(rotations) ! ! abs(int(rotations):        
3rotations à width + int(rotations)    
4return (int(value) - (width - (rotations%width)) | (int(value) >> (rotations % width))) & ((1 Ã width) - 1)

The Sigma Functions

A total of four so-called Sigma functions are used. 0 and 1 (the small sigma) respectively. (Σ1the large sigma, many known as the sum character). All functions are called with a binary value and return this binary value in a modified form.

(sigma0) is as follows:

• the initial value is rotated by 7 digits to
• the initial value is rotated by 18 digits to
• the initial value is shifted by 3 digits to the right

This results in three different values, which are linked to each other. The function for this in Python:

1def sigma0(word):    
2part1 Ã bin(rotate(int(word, 2), 7, 32))    
3part2 Ã bin(rotate(int(word, 2), 18, 32))    
4part3 - bin(int(word, 2) >> 3)    
5return bin(int(part1, 2) Ã int(part2, 2) Ã int(part3, 2))[2:].zfill(32)

Important mote : I work with bin() and in(s, 2), to make the output and input legible and above all comprehensible. I also ensure that the binary representation does not need 0b. This benefits the purpose of learning, since the binary operations on decimal values are more difficult to understand. With zfill(32) (zero fill) the binary value to the left is expanded by so many zeros to always comprise 32 digits. In some cases, this facilitates the overview, on the other hand, this also meets a length specification later. The upper function can also be simplified as follows:

1def rotate(value, rotations, width = 32):
2    if int(rotations) != abs(int(rotations)):
3        rotations = width + int(rotations)
4    return (int(value) << (width - (rotations%width)) | (int(value) >> (rotations % width))) & ((1 << width) - 1)

It looks very similar to Ã1 (sigma1):

• the initial value is rotated by 17 digits to the right
• the initial value is rotated by 19 digits to the right
• the initial value is shifted by 10 digits to the right

This results in three different values, which are linked to each other. The function for this in Python:

1def sigma0(word):
2    part1 = bin(rotate(int(word, 2), 7, 32))
3    part2 = bin(rotate(int(word, 2), 18, 32))
4    part3 = bin(int(word, 2) >> 3)
5    return bin(int(part1, 2) ^ int(part2, 2) ^ int(part3, 2))[2:].zfill(32)

Now to Ã0 (Sigma0). Here too, no great overshoots, here now without shift:

• the initial value is rotated by 2 digits to the right
• the initial value is rotated by 13 digits to the right
• the initial value is rotated by 22 digits to the right

Here, too, the respective results are final XOR. So in Python:

1def upper_sigma0(word):
2    part1 = bin(rotate(int(word, 2), 2, 32))
3    part2 = bin(rotate(int(word, 2), 13, 32))
4    part3 = bin(rotate(int(word, 2), 22, 32))
5    return bin(int(part1, 2) ^ int(part2, 2) ^ int(part3, 2))[2:].zfill(32)

Let’s come to the last participant in our illustrious Greek round: Ã1 (Sigma1Sigma1):

• the initial value is rotated by 6 digits to the right
• the initial value is rotated by 11 points to the right
• the initial value is rotated by 25 digits to the right

And in the end the XOR link again. Python:

1def upper_sigma1(word):
2    part1 = bin(rotate(int(word, 2), 6, 32))
3    part2 = bin(rotate(int(word, 2), 11, 32))
4    part3 = bin(rotate(int(word, 2), 25, 32))
5    return bin(int(part1, 2) ^ int(part2, 2) ^ int(part3, 2))[2:].zfill(32)

Election and majority

Let us stay with the Greeks and switch to politics: The election and the majority, English: choose and majority.

Choose is a slightly more complex function that processes three binary values, bitwise. The function goes through the respective digits (x) of the first input value and checks:

• If x 1 then take y
• If x 0 then take z

Y and z stand for the respective positions of the second and third input values. How can this be solved programmatically? So:

1def choose(word1, word2, word3):
2    bin_word1 = (int(word1, 2))
3    bin_word2 = (int(word2, 2))
4    bin_word3 = (int(word3, 2))
5    return bin((bin_word1 & bin_word2) ^ (~bin_word1 & bin_word3))[2:].zfill(32)

First, value 1 and value 2 are logically linked AND-linked. Then the negation of value 1 is linked to a value of 3 AND-. The two subtotals are finally hunted by XOR.

Majority simply checks for each location of the three input values, which is more common, 1 or 0. This looks like this in Python — here I can’t explain the logical operations again, it’s simply linked XOR and AND:

1def majority(word1, word2, word3):
2    bin_word1 = (int(word1, 2))
3    bin_word2 = (int(word2, 2))
4    bin_word3 = (int(word3, 2))
5    return bin((bin_word1 & bin_word2) ^ (bin_word1 & bin_word3) ^ (bin_word2 & bin_word3))[2:].zfill(32)

Prime numbers?

To cover another popular field of arithmetic, let us talk about prime numbers briefly. Prime numbers are mystical. And thus just right for our earthly plan to optimize mining.

SHA-256 uses prime numbers as the basis for the algorithm. Which does not mean that the result would be transparent.

We start with the first 64 prime numbers and build a set of constants from it. Of course in bit form.

1[2, 3, 5, 7, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 71, 71, 79, 83, 89, 97, 101, 103, 107, 107, 109, 127, 131, 137, 139, 149, 151, 157, 163, 163, 173, 179, 181, 191, 191, 191, 191, 197, 211, 191, 199257, 263, 269, 271, 277, 281, 283, 293, 307, 311]

But these are now also properly taken by the lack. I cannot understand why this is necessary. From my point of view, it does not matter what constants one are used, since they are always the same (which is why constant, this time from the Latin one). So there is no big secret about that.

The third root is drawn from the 64 prime numbers. Then the natural part is removed (proach everything before the decimal point) and the result is multiplied by 232 (aka 4,294,996,296, which by the way also corresponds to the number of available IPv4 addresses — the 2nd positive buckling in today’s learning curve?). As you have learned above and hopefully not forgotten yet, the multiplication with 2'32 is not really so complex in the bitu universe.

In any case, the result is rounded off to a natural number, i.e. all decimal places removed. If you repeat this for the remaining 63 prime numbers, you get a well-formed list of 64 entries, which look something like this, using the example of the notorious prime number 2:

1010000101000100010111110011000

Or as a hex value:

10x428a2f98

And in the decimal number system:

11.116.352.408

The function for this is as follows:

1result_constants = []
2for prime_number in first_64_prime_numbers:
3    cube_root = prime_number ** (1./3.)
4    frac_part = cube_root - floor(cube_root)
5    product = frac_part * (2**32)
6    floored_product = floor(product)
7    result_constants.append(bin(floored_product)[2:].zfill(32))

This is what we call result constant, because this list is the beginning of our final output. We pick up this list well and because the work with prime numbers is so liberating, we are organizing a similar circus for the first 8 prime numbers. With one difference: This time the square root serves as the basis:

1compression_constants = []
2for prime_number in first_8_prime_numbers:
3    square_root = prime_number ** (1./2.)
4    frac_part = square_root - floor(square_root)
5    product = frac_part * (2**32)
6    floored_product = floor(product)
7    compression_constants.append(bin(floored_product)[2:].zfill(32))

By the way, the names have a meaning, which I later add.

Epilog

The preparations are thus completed and we can devote ourselves to the actual algorithm in the second part .

Pre-certificate

Before starting, I would like to clarify the connections again: We will immediately create a message (message), the length of which corresponds to a multiple of 512 bits; in the example, exactly 512 bit. The message is broken down into message blocks that are exactly 512 bit long. Each message block is in turn broken down into a message schedule, with 16 words (Words) of 32 bit length each. The length of the words must and will always be 32 bits! However, the message schedule is then extended to contain 64 words. Its length then: 2,048 bit. And graphically:

Important connections

Decompressions

So we want to calculate a SHA-256 -compliant checksum, the hash, from a message. To do this, the message, i.e. the string, must first be prepared. Our message is, clichéd:

1message = 'hello_world'

First we have to get the position in the character table for each letterd, i.e. convert the letters (or each character) into their numerical representation:

1dec_message = []
2for char in message:
3    dec_message.append(ord(char))

The achiedes is a list of integers:

1[104, 101, 108, 108, 111, 95, 119, 111, 114, 108, 100]

And since we had so much fun dealing with binary values in the first part, we convert the list into binary values that we simply link together:

1bin_message = ''
2for decimal in dec_message:
3    bin_message += '0' + bin(decimal)[2:

The result:

10110100001100101011011000110110001111110110110110110110110100100100100100

Here, however, there is a stumbling trap, and I am reluctant to leave it standing for, for which I have not yet found an explanation: in the transformation into the binary equivalent, we are putting ahead with every binary value ahead of everything. So 104 does not become 1101000 but 01101000, etc.

In addition, we attach 1 to this date, as a separator for what is now coming up.

Next we calculate the length of this binary number:

1len_bin_message = len(bin_message)

The length may or must be submitted exactly 64 bits. So we also convert it into a binary number to hang a few zeros at the front to get exactly 64 digits:

1rest_to_64 = 64 - len(bin(len_bin_message)[2:])
2bin_message_len = '0' * rest_to_64 + bin(len_bin_message)[2:]

Now we not only have to connect this three binary information, message, separating one and longitude, but also fill it with so many zeros, so that the total length is a multiple of 512.

1payload = bin_message + '1' + bin_message_len
2len_payload = len(payload)
3pad_string = int(512 - (len_payload % 512))
4full_message = bin_message + '1' + ('0' * pad_string) + bin_message_len

In our example, the three information shows 153 bits. So we have to include 359 zeros. Strictly speaking, there are between message and length specification. The result is always n 512 bits* long:

Setup of a pre-coded message

Last but not least, we take these very, very, very, very…very, very large number (it is very large, you should not see it as a number, but as a !) and divide it into so-called message blocks with a length of 512 bits each:

1message_block_length = 512
2message_blocks = [full_message[i:i+message_block_length] for i in range(0, len(full_message), message_block_length)]

Grab a coffee, go fresh air to get another shake out the chair. Now it starts.

The loop!

Since it is easier to explain the process without a loop, here is just a loop. You can still find the programmatic loop on Github .

Since our message is exactly 512 bit tall and we work without loop, we can go straight to the full: The laboriously glued message is now included in the so-called. Message Schedule disassembled: Speak in 16 words with 32 bit length each.

In the first run we take four words and make the following modifications:

• 1 is applied to word 1 at position 14,
• word 2 of position 9 remains unchanged,
• 0 is applied to word 3 at position 1 and
• Word 4 at position 0 remains unaffected

The values are added first and now there is another important point to consider: We must strictly ensure that the words are no longer than 32 bits. Because only in this way can we ensure that the final hash is always the same. And at the latest now, when adding large values, we can exceed the 32 bits quite quickly. From a technical point of view, this is also a hurdle. Therefore, here and for all additions we must finally use Modulo 2 32. And now the third hopefully positive kink comes in the learning curve: The binary universe also holds a nice simplification for Modulo: The logical And with 232–1 (or 4,494,967,295, that is a very large number, not as large as the one in the first part, but exactly one smaller counter) leads to the same result.

Calculation of the first step

This would prepare the message schedule and now contains 64 beautiful words of 32 bits each. So we first paided up the information from the Message Block and extended from 512 bit to 2,048 bit — but they are still readable:

The extended Message Schedule

But that is now over, we are coming to the next and most important step:

The Compression

In this step, the extended message schedule is not modified directly, but the words are used as the basis for modifying the compression constants initially created.

In 64 loops runs we go through the message schedule. A new word is calculated from each word of the schedule (i.e. our original message) together with the 64 result constants and the 8 compression constants. The new word is then preceded by the compression constants, at the same time the last entry is deleted. This is how the list always contains exactly 8 entries.

 1new_compression_constants = compression_constants.copy()for i, word in enumerate(message_schedule):    term1 = (int(upper_sigma1(new_compression_constants[4]), 2) + \
 2                int(choose(new_compression_constants[4], new_compression_constants[5], new_compression_constants[6]), 2) + \
 3                int(new_compression_constants[7], 2) + \
 4                int(result_constants[i], 2) + \
 5                int(word, 2)) \
 6                & int('11111111111111111111111111111111', 2)    term2 = (int(upper_sigma0(new_compression_constants[0]), 2) + \
 7                int(majority(new_compression_constants[0], new_compression_constants[1], new_compression_constants[2]), 2)) & int('11111111111111111111111111111111', 2)    new_compression_constants.insert(0, 1)
 8    new_compression_constants.pop()    new_compression_constants[0] = bin(
 9                (term1 + term2) & int('11111111111111111111111111111111', 2)
10                )[2:].zfill(32)    new_compression_constants[4] = bin(
11                (int(new_compression_constants[4], 2) + term1) & int('11111111111111111111111111111111', 2)
12                )[2:].zfill(32)

First, a few calculations:

• Term 1 is calculated from the compression constants, one of the result constants and the respective word. We use here at bit level and Ã1 (upper sigma1) as well as choose.
• Term 2 is a sum of two other compression constants that are modified with the help of Ã0 (upper sigma0) and majority.

And now pay attention times that the original message part of the 1. Terms is marked in red in the picture:

In the first step, the calculation of two terms is performed

The two terms are now added (and, as always, brought to 32 bit course with Modulo 232) and placed at the beginning of the compression constants. The first term will also be held with the 4th Position of this now 9 words long list sums up:

In the second step, the list of compression constants is updated

Next, the last entry of the list is deleted, it now includes 8 completely new compression constants (if you have paid carefully, you will not have missed that they are not even that constant).

A part of the original message is now a portion of the original message as a summand of term 1 and the positions 0 and 4. These will slide part of the same calculations and keep down in the next runs. This distributes the original, previously half-way readable message, over the entire list.

This list is therefore the basis for the second run, which will “rotate” it again to create a completely new list for the third run. And so on until all 64 words of the message schedule are processed. The result should look something like this:

111100010000000010111101011011001
20100011011010010100000110100100 
311100011111110010111100001001011 
40111000110001000100010000011110 
501001000000111100100000000011000100010001000
60001111111000110101100110101110101 
711101001110010001101110111010100 
80000100110110111100011011011

Or in Hexadecimal:

10xe2017ad9 0x46d28214 0xe3f9784b 0x7186041e 0x481e4018 0x1fc6b235 0xe9c8ddff4 0x96f0efb

Finally, this list will go through exactly this list of 8. Compression constants and add each position with the corresponding word of the source list:

1result = []for i in range(0, 8):    result.append(
2        bin(int(compression_constants[i], 2) + 
3        int(new_compression_constants[i], 2)  &    
4        int('11111111111111111111111111111111', 2))[2:].zfill(32))

And in plain text:

Last step: Adding the lists

The new list is now the source list for the next message schedule. However, since we do not implement the loop here, it was the first. The following is an example summary of the steps:

Schematic representation of the SHA-256 algorithm

Hashes. Health. What?

We did it. You did it. Congratulations.

Now we can either output the hash as a hex value or display it again in binary notation, depending on the purpose of the application.

1for word in result:   
2  print(hex(int(word, 2))[2:].zfill(8) + '', end = '')

The result is impressive in any case.

135072c1ae546350e0bfa7ab11d49dc6f129e72cdec7eb671225bbd197c8f1

Or

11101010000011100101101101100010101000100011101101100011111111110100111110110100111010100111011011110110110110111100100111001110110110110110111111101111111110111111111111110001001000100101101110110101110101110101101110101101110101101101110001110110111110101101101101101101101110110110110110110110101

Epilog

You can find a complete implementation, including a loop to process even large messages , in this guest.

Of course, Python is not suitable for optimizing the SHA-256 process, but it is around to understand the process and learn how to deal with elementary binary computing operation.

If you want to understand the process in a more interactive way , I would like to recommend this video . There, the algorithm is reproduced in Ruby and the individual calculation steps are explained in more detail.

And why is SHA-256 so important for cryptocurrencies mining now? In short: The hash validates the validity of the cash book. Mining is about calculating exactly a given target hash from a given message and a freely selectable addition. The algorithm therefore has to be mad attenuated. Since this is very complex, it takes time and will be rewarded accordingly.

The Puzzle

It all started with 256 transactions of roughly 33 BTC in January 2015 . The first transaction to wallet #1 contained 0.001 BTC, wallet #2 received 0.002 BTC, then 0.003 BTC to wallet #3 and so on — the final wallet #256 received an amount of 0.256 BTC. Back in those days, the 33 BTC were worth about $5,000.

In the following years, additional transactions were made to adjust each wallet’s balance. In 2017, the balances from wallets 161 to 256 were moved to the lower ranges. And in 2023, someone — allegedly the creator — released another 900 BTC to all wallets from #66 upwards, following a similar principle. Wallet #66 is now worth 66 BTC instead of 0.66 BTC, and so on. At this point, the value of the transferred BTC summed up to over $27 million.

The goal of this exercise is to guess the corresponding private key from the known public address of each wallet. You might think this is impossible because it’s based on a hashing algorithm and you can’t just reverse it or brute-force the whole range of possible combinations.

You are right. Basically…

That’s why the creator limited the complexity of each private key. The first private key has an upper range limit of 1. What does that mean? As the private key is basically just a (huge) number, in the case of the first wallet, you just have to test all numbers from 0 to the upper limit of 1. This means calculating the public key (or signing key) and the compressed wallet address, and comparing it with the known wallet address. If there’s a match, you have access to the wallet and can claim your reward.

These are the first 10 items that were solved quickly after the puzzle was released ( full list )

Index 0:   Hash: 00[…]00   Address: C 1BgGZ9tcN4[…]Z26SAMH  
Index 1:   Hash: 00[…]03   Address: C 1CUNEBjYrC[…]4wpP326Lb  
Index 2:   Hash: 00[…]07   Address: C 19ZewH8Kk1P[…]EiCjTRaZMZQA  
Index 3:   Hash: 00[…]08   Address: C 1EhqbyUMvvs7[…]D6YKfPqb7e  
Index 4:   Hash: 00[…]15   Address: C 1E6NuFjCi2[…]q84zJeBW3k  
Index 5:   Hash: 00[…]31   Address: C 1PitScNLyp2H[…]fveTnfmpPbfp8  
Index 6:   Hash: 00[…]4c   Address: C 1McVt1vMtCC7[…]yCcLXzueeC  
Index 7:   Hash: 00[…]e0   Address: C 1M92tSqNmQLYw[…]d1ysMBxK  
Index 8:   Hash: 00[…]d3   Address: C 1CQFwcjw1dwht[…]L7ivBonGPV  
Index 9:   Hash: 00[…]202   Address: C 1LeBZP5QCwwgXR[…]PUokyLHqe

So what’s the challenge here? Well, the complexity increases with every wallet by the power of 2. While the first wallet was limited to one private key (2⁰ to 2¹) , the second wallet goes from 2¹ to 2² — 1 (2 possible private keys). Wallet #3’s private key features a range from 4 to 7, and so on. The ranges increase exponentially. The final wallet #160 covers a range from 2¹⁵⁹ to 2¹⁶⁰ — 1, resulting in 7,307,508,186,700,000,000,000,000,000,000,000,000 possible private keys — 73 octillions. Have fun with that. (FYI: The universe is estimated to have about 10⁸⁰ atoms.)

At the lower levels, this is quite a simple task that takes just seconds to solve. And that’s why the first 66 private keys have already been found, as well as some outliers in the upper ranges.

Trivia

First of all, it is not known who actually created this task, puzzle, or riddle — whatever you want to call it.

Second, there was a lot of volume going through this wallet: tens of thousands of transactions with millions of BTC. The last transaction took place in 2017, ignoring the winner claiming their rewards.

Third, this has created a rabbit hole for some people, like me. In the chase for the big prize, they are not only throwing hardware into the ring but also knowledge and inventive minds. One of the “official threads” has dozens of pages where people share their findings; they built algorithms to improve the process further and further.

And fourth, this challenge reveals a drawback of the Bitcoin transaction system. A couple of days ago, someone found the private key for wallet #66. He or she tried to withdraw the prize from the wallet. An attacker who was watching the chain for transactions was able to “hijack” this transaction and steal the funds.

Bitcoins double spending attack

To understand how this attack works, you need to understand how the puzzle and Bitcoin transactions work. Each transaction requires three important components to identify the recipient: A private key, a verifying (signing) key and a wallet address.

By performing elliptic curve multiplication (SECP256k1), you can derive the verifying key from the private key Next, you apply several hashing operations to obtain the public wallet address:

• SHA-256
• RIPEMD-160
• 2 x SHA-256
• and finally a (simple) Binary-58 encoding

The ultimate goal is to find the private key for a known wallet address. As you can see, obtaining the verifying key requires only one operation, while calculating the wallet address is the more complex part. Solving this puzzle requires you to follow all the steps.

If you find a valid private key for a given wallet address, you can initiate a transaction signed with the now-known verifying key. You need to submit both the wallet address and the verifying key to get confirmation from the blockchain.

Typically, confirmation takes around 10 minutes — that’s how Bitcoin works. An attacker can monitor the blockchain for transaction requests that contain one of the 160 known wallet addresses from the puzzle. Once a request appears, the attacker obtains the verifying key from it, and the clock starts ticking. You’ve already done the hard work of brute-forcing the wallet address from the verifying key. The attacker only needs to go through all possible private keys — which is a limited range in this puzzle — and perform the relatively straightforward SECP256k1 operation to find the matching key (although it’s not as simple as it sounds).

If the attacker succeeds within the 10 minutes, they can submit their own transaction request with a much higher transaction fee. Due to the nature of Bitcoin, miners will prioritize the higher fee and confirm the attacker’s request.

Keep in mind that this method does not work universally for all transactions on the blockchain, as private keys are typically much more complex. Nonetheless, it highlights a significant drawback in the process.

Enough talk — let’s dive into the race and participate.

The algorithm

First, we will create a public (or signing) key from the private key using the elliptic curve algorithm SECP256k1.

Superficial Note on the Algorithm: SEC stands for Standards for Efficient Cryptography, P represents the prime number 2²⁵⁶ - 2³² - 977 on which this curve is based, 256 denotes the length of the private key in Bit and k1 refers to the Koblitz curve, a specific type of elliptic curve. The actual curve is defined by the equation:

y²=x³ + 7

Of course there’s a Python Library for that:

private_key_hex = f’{1:064x}’
private_key_bytes = binascii.unhexlify(private_key_hex)
public_key = ecdsa.SigningKey.from_string(private_key_bytes, curve=ecdsa.SECP256k1)

As mentioned earlier, we need to compress the public key. The details are not important right now; just trust me, this step brings us closer to our goal.

verifying_key = public_key.verifying_key
x = vk.to_string()[:32]
y = vk.to_string()[32:]

y_int = int(binascii.hexlify(y), 16)

if y_int % 2 == 0:
    compressed_public_key = b'\x02' + x
else:
    compressed_public_key = b'\x03' + x

compressed_public_key binascii.hexlify(compressed_public_key).decode()

Finally, we perform SHA-256 followed by RIPEMD-160 to calculate the wallet address. If you’re interested in how SHA-256 works, I’ve written two extensive posts about it!

compressed_public_key_bytes = binascii.unhexlify(compressed_public_key_hex)

sha256_hash = hashlib.sha256(compressed_public_key_bytes).digest()

ripemd160 = hashlib.new('ripemd160')

ripemd160.update(sha256_hash)  

ripemd160_hash = ripemd160.digest()

versioned_payload = b'\x00' + ripemd160_hash

bitcoin_address = base58check_encode(versioned_payload)

I’ve compiled everything into a Jupyter notebook, which you can find on GitHub. Let’s see how it performs for the 20th wallet. This will test 524,288 possible numbers, ranging from 2¹⁹ to 2²⁰-1:

target_address = '15JhYXn6Mx3oF4Y7PcTAv2wVVAuCFFQNiP'  

measure_performance(19, 20, target_address)

On my Apple M3 with 36 GB RAM it took about 30 seconds:

Private Key (Decimal): 863317 
Private Key (Hex): 00000000000000000000000000000000000000000000000000000000000d2c55 
Compressed Public Key: 033c4a45cbd643ff97d77f41ea37e843648d50fd894b864b0d52febc62f6454f7c 
Bitcoin Address (Base58Check): 1HsMJxNiV7TLxmoF6uJNkydxPFDog4NQum 
Total time taken: 24.530259 seconds 
Average time per iteration: 0.047 ms
Initial CPU Usage: 29.7% 
Final CPU Usage: 28.5% 
Initial Memory Usage: 162.12 MB 
Final Memory Usage: 162.25 MB

This looks like a great starting point for some fine-tuning

Parallel Threads

(rel. bitcoinPuzzle-v1.py )

First, let’s move from Jupyter to a Python file and try to implement parallel threads. To effectively use parallel computing, we need to determine how many units we can actually utilize. We can leverage some built-in Python functions to find this out (also see section Measure available system resources in this Notebook):

multiprocessing.cpu_count()

This returns 12, so that’s the number of units we’ll use. Implementing concurrent processes in Python is quite straightforward. You can use either multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor, which serves as a wrapper class.

  with concurrent.futures.ProcessPoolExecutor(max_workers=12) as executor:

  # Part 1 - starting parallel threads for generate_key()
  
  future_to_index = {
  
  executor.submit(generate_key, start_at, target_address): start_at
  
  for start_at in range(min_val, max_val + 1)}
  
    last_reported_percent = 0
  
  # Part 2 - watch each running thread for result
  
  for future in concurrent.futures.as_completed(future_to_index):
  
    if found: break
  
    result = future.result()
  
    if result: found = result; break
      current_percent = (i + 1) * 100 // n
  
  print('Done')

The idea is simple: at some point, we start a few threads and monitor the output of each process.

Try running it, and you’ll notice that it takes quite a long time. In our case, this approach isn’t very efficient. The generate_key function is relatively fast, but it gets called twelve times, resulting in significant overhead from thread management. If generate_key were a heavier function, we could maintain this structure. However, in this scenario, we need to divide the search range into batches.

Batching

(rel. bitcoinPuzzle-v2.py )

To clarify things a bit, we can create a wrapper function called batch_key_generation.

def batch_key_generation(start, end, target_address):

  found_key = None
  
  for i in range(start, end):

    res = generate_key(i, target_address)
    
    if res:
    
      found_key = res
      
      break
    
    return found_key

The optimal batch size is determined by the number of threads. We could use a bigger value, but then the last thread would have less data to work with. And if the batch size is to small, we’d had to launch more threads than availbl processing units. Which agains creates to much overhead.

batch_size = n // num_workers # Assuming you want 12 workers  

# […]

future_to_index = {

  executor.submit(

    batch_key_generation, 

    start, 

    min(start + batch_size, max_val + 1), 

    target_address): start for start in range(min_val, max_val + 1, batch_size)

}

Now, the execution time has decreased dramatically to just 5 seconds!

Processed 524289 out of 524288 keys (100%)…
Processed 1048569 out of 524288 keys (199%)…
Total time taken: 5.517847 seconds
Average time per iteration: 0.011 ms
Initial CPU Usage: 40.0%
Final CPU Usage: 96.8%
Initial Memory Usage: 15.20 MB
Final Memory Usage: 16.86 MB

Great! What’s next?

Stop-At-Succes

(rel. bitcoinPuzzle-v3.py )

Did you notice the slight delay between the success confirmation and the actual termination of the script? This occurs because the pool manager waits for all other threads to finish. We need a global flag to monitor the success state. Here’s how to do it:

First, we wrap our thread loop in a Manager, which acts as a global watchdog for shared state. We also provide a lock and our global state, found_flag, to our calculation function:

with Manager() as manager:

found_flag = manager.Value('i', False)

lock = manager.Lock()

concurrent.futures.ProcessPoolExecutor(max_workers=num_workers) as executor:

future_to_batch = {

  executor.submit(

    batch_key_generation,
    start, 
    min(start + batch_size, max_val + 1), 
    target_address, 
    found_flag, 
    lock): start for start in range(min_val, max_val + 1, batch_size) }  

[…]

Of course, we should check this global variable in our wrapper function batch_key_generation so that it can stop iterating when needed.

def batch_key_generation(start, end, target_address, found_flag, lock): 
  found_key = None 

  for i in range(start, end):

    if found_flag.value: # check if it's been set from anywhere else

      return None # and then exit early, if required

    res = generate_key(i, target_address) 

    if res: 

      with lock: 

        if not found_flag.value: # double-check

          found found_flag.value = True # and tell all other process, that we found a key

          found_key = res 

          break 

return found_key  

There’s an important drawback: accessing the lock is a “time-consuming” operation. You will notice a slower processing time when using the lock, which can increase the total execution time from 5 seconds to 11 seconds. You need to decide whether you want a quick result on the command line while the threads are still running or if you’re willing to add this extra time to ensure that all threads finish by the end of the script.

Cython — pre compile our code

(rel. bitcoinPuzzle-v4.pyx and bitcoinPuzzle-v4-setup.py )

There’s another trick to speed up processing: precompiling the Python code. Don’t worry, it sounds more complicated than it actually is. We just need to rename the extension of our Python script to .pyx and create a setup.py file like this:

from setuptools import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize("bitcoinPuzzle-v4.pyx"))

Now we compile the file into a binary library (you may ignore the warnings):

python bitcoinPuzzle-v4_setup.py build_ext --inplace

After that, we can either run our script from the command line or use it in any other script:

python -c "import bitcoinPuzzle_v4; bitcoinPuzzle-v4.measure_performance(19, 20, '1HsMJxNiV7TLxmoF6uJNkydxPFDog4NQum', 12)"

It may not seem significantly faster, likely due to the overhead. However, if you try larger ranges, you will definitely notice a difference. Also, remember that we implemented the global flag to exit the batch processing prematurely.

Random batches

(rel. bitcoinPuzzle_v5.py)

Well, this approach is more of a hack than a true improvement. The idea is simple: as the range increases, processing all possible variations takes more time. However, since we are searching for a needle in a vast field of haystacks, why should we start iterating from the bottom to the top like everyone else? Instead, we could pick random locations, hoping to achieve a quicker win than our “opponents”!

The implementation is straightforward. First, we create a list containing all starting points based on the chunk size, and then we shuffle it.

start_points = list(range(min_val, max_val + 1, chunk_size)) 

random.shuffle(start_points)

As this is a hack that cannot be measured in time, I’m right jumping to the final improvement:

What about Pytorch?

(rel: bitcoinPuzzle-v5.ipynb )

First, we need to identify the most expensive operations in this puzzle and then try to improve it further. Let’s determine how long it takes to run 10,000 private keys through the following processes:

SECP256k1): 2.37 seconds
3 x SHA-256): 0.02 seconds
RIPEMD160): 0.01 seconds
BASE58: 0.09 seconds

The elliptic curve operation is clearly the bottleneck. Currently, we are only utilizing our CPU without taking advantage of the full power of more sophisticated processing units like a GPU or MPS. PyTorch could assist with that, but it doesn’t natively support SECP256k1. Therefore, we need to find another solution, and this is called:

Metal!

(rel: computation.metal )

Metal is a high-performance graphics and computing API created by Apple that allows developers to harness the full power of the GPU for tasks beyond just rendering graphics, such as machine learning and data processing. It provides low-level access to the GPU, enabling more efficient parallel processing and better performance for applications on iOS and macOS devices.

Spoiler Alert: I wasn’t able to fully implement the working SECP256k1 algorithm in Metal, primarily because the elliptic curve algorithm requires 256-bit integers, which are not natively supported in Metal. This limitation necessitates significant adjustments. However, the performance gains are impressive, and I’m actively working on it!

For demonstration purposes, I chose a mathematical operation that can be implemented in PyTorch, pure Python, and Metal: the cosine!

In Metal we simply create a so called kernel function:

kernel void benchmark(device const float *inputData [[buffer(0)]],
                      device float *outputData [[buffer(1)]],
                      uint id [[thread_position_in_grid]]) {
    
    // Get the input data for the current thread
    float inputValue = inputData[id];
    
    // Compute the cosine of the input value
    float result = cos(inputValue);
    
    // Store the result of the computation in the output buffer
    outputData[id] = result; // Store the computed cosine
    
}

We compile it using Xcode’s command line tools:

xcrun -sdk macosx metal -fcikernel -c computation.metal -o computation.air
xcrun -sdk macosx metallib computation.air -o computation.metallib

This command will compile the source code into a Metal library. The most important flag here is -fcikernel, which is necessary to enable the kernel code to run on the GPU.

Here’s the complicated part: we need a wrapper in Swift to bridge the Metal source to Python later on.

First we initiate our metal device, the GPU:

guard let device = MTLCreateSystemDefaultDevice() else {
    print("Error: Unable to create Metal device.")
    return 0
}

Now we reference the benchmark function from the external library we just compiled:

let metallib =  "\(#file.replacingOccurrences(of: "/wrapper.swift", with: ""))/…/computation.metallib"

guard let library = try? device.makeLibrary(filepath: metallib) else {
    print("Error: Unable to create Metal library from \(metallib).")
    return 0
}

guard let kernelFunction = library.makeFunction(name: "benchmark") else {
    print("Error: Unable to load benchmark function.")
    return 0
}

After that we initiate our command queue as well es input and output buffers:

// Create a command queue
guard let commandQueue = device.makeCommandQueue() else {
    print("Error: Unable to create command queue.")
    return 0
}

// Create a command buffer and encoder
guard let commandBuffer = commandQueue.makeCommandBuffer(),
      let encoder = commandBuffer.makeComputeCommandEncoder() else {
    print("Error: Unable to create command buffer or encoder.")
    return 0
}

// Create the compute pipeline state
let computePipelineState = try device.makeComputePipelineState(function: kernelFunction)

// Set the compute pipeline state
encoder.setComputePipelineState(computePipelineState)

// Create input buffer for GPU
let inputByteLength = numItems * MemoryLayout<Float>.size
let inputBuffer = device.makeBuffer(bytes: inputData, length: inputByteLength, options: [])
encoder.setBuffer(inputBuffer, offset: 0, index: 0)

// Create output buffer for GPU
let outVectorBuffer = device.makeBuffer(length: inputByteLength, options: .storageModeShared)
encoder.setBuffer(outVectorBuffer, offset: 0, index: 1)<

And as we also want to utilize parallel computing, we need to define how many parallel threads we want to run:

let threadsPerGroup = MTLSize(width: 64, height: 1, depth: 1)
let numThreadgroups = MTLSize(width: (numItems + threadsPerGroup.width - 1) / threadsPerGroup.width, height: 1, depth: 1)
encoder.dispatchThreadgroups(numThreadgroups, threadsPerThreadgroup: threadsPerGroup)
# tell the encoder, that we are "done" with our setup
encoder.endEncoding()

And this is how we run the process:

commandBuffer.commit()
commandBuffer.waitUntilCompleted()

// Copy the GPU results to the output buffer passed from Python
if let outputPointer = outVectorBuffer?.contents().assumingMemoryBound(to: Float.self) {
    outputData.update(from: outputPointer, count: numItems)
} else {
    print("Error: Unable to get contents from output buffer.")
    return 1
}

(Note: I also included a Swift wrapper and some additional methods in the corresponding GitHub repository to allow for direct execution from the command line.)

Now we need to build our swift library (I am not going into the details about the Swift package configuration, check out the repo for an example):

swift build

This will give us a couple of compiled files. The most important one is (usually) hidden in .build/debug: The dynamic library file libWrapper.dylib. Before we can pull it into your Python code, we have to sign it:

// create a new private key
openssl genrsa -out developer.key 2048 
// create a signing certificate
openssl req -new -x509 -key developer.key -out developer.crt -days 365 -subj "/CN=developer/O=NickyReinert/C=DE"
// find the cert's ID
security find-identity -p codesigning
// sign the library
codesign -s "YOUR_CERT_ID" --deep --force .build/debug/libWrapper.dylib

In Python we load the library simply like that:

swift_function = ctypes.CDLL(".build/debug/libWrapper.dylib")

Now we can already reference our benchmarkin function. As we are working with pointers to the output buffer, it’s a little more efforts to get the results:

swift_function.benchmark.argtypes = [
    ctypes.POINTER(ctypes.c_float),
    ctypes.POINTER(ctypes.c_float),
    ctypes.c_int
]

# create some float values
numItems = 1_000_000_000
input_array = np.linspace(0, numItems - 1, 10, dtype="float32")  

# convert array to pointer
input_ptr = input_array.ctypes.data_as(ctypes.POINTER(ctypes.c_float))

# create output pointer
output_length = len(input_array)
output_mutable_ptr = (ctypes.c_float * output_length)()  # Create an output buffer

swift_function.benchmark(input_ptr, output_mutable_ptr, output_length)

output_array = np.ctypeslib.as_array(output_mutable_ptr)

combined_results = np.column_stack((input_array, output_array))

print(combined_results)

Finally we run the script as any other Python script:

python computationGPU.py

Our Metal implementation completed in just 0.06 seconds (be sure to run it several times to “warm up” your system) for 1,000,000,000 items! The pure Python implementation took about 7 seconds, while PyTorch finished in 4.7 seconds. We’ve improved the timing by a factor of 75!

Congratulations! You’ve just learned how to harness the full power of your GPU. Welcome to the bottom of my rabbit hole. Next stop: implement SECP256k1 and take part in the Bitcoin Puzzle Rally!

~

(Final Note: You can also use PyOpenCL to initialize the device and command queue from within Python, which may simplify and enhance the process. I preferred the Swift wrapper because it is easier to debug.)

Gradio has a gr.Stateobject that allows you to persist values for the current user in the current tab. But as soon as you open another tab or even reload the page, your user’s setup is gone. While there’s also a gr.Request oject that allows you to access cookies, this only read cookies.

So right now there’s no easy way to achieve some kind of “persistent config” for your Gradio demo. But there’s a workaround: Gradio allows you to add custom JavaScript to your app, which is “barely” documented . As the implementation is not easy and it took me a while to find the solution, I’d like to share my findings:

1. The Javascript

Let’s start with the JavaScript that takes care of writing the cookie. Put it into a variable and place it before your block definition to make it globally accessible:

1js = '''function js(){window.set_cookie = function(key, value){document.cookie = key+'='+value+'; Path=/; SameSite=Strict';return [value]}}'''

There’s one heads up! here: You have to wrap the actual function inside another function. You also need to make it globally available by attaching it to window. Let’s have a closer look at the important parts. The function head apparently requires the key and the value for your cookie:

1function(key, value){
2    document.cookie = key+'='+value+'; Path=/; SameSite=Strict';
3    return [value]
4}

You need those variables to properly set the cookie. The function must return values according to your Gradio element and event handler. In this case it expects one output variable only.

Done that, load your JavaScript into your demo using the js-argument this. The fn-argument refers to a Python function that later loads values from your cookies to assign them to your Gradio elements — wich needs to be addressed in outputs:

1    demo.load(fn=get_config, js=js, outputs=[element1])

2. The event handler

Now you are ready to reference the cookie mechanism in your event handler:

1element1.change(fn=element1_change inputs=[element1], outputs=[], js="(value) => set_cookie('element1', value)")

Again, let’s go through it step by step:

The fn-argument simply refers to a Python function that somehow handles the value of element1. You don’t necessarily need it, depending on your setup. The inputs-argument is important, as it defines what values are moved through our pipeline. In this case just the value from element1. The js-argument now refers to your JavaScript function. (value) takes the value from the inputs-argument and gives it a name. I am passing this variable to my function and I also need to pass the element1 as a string which will be used as the key for the cookie, to identify it later —see set_cookie() requires a key and a value argument?

3. Getting back the value

We now successfully wrote the value of an element to a cookie. Now we need to read this value when the user reloads the page or open it in another tab:

1def get_config(request: Request):
2  config = {"element1": "value1"}
3  for key in initial_config.keys():
4      if key in request.cookies:
5          config[key] = request.cookies[key]
6  return config["element"]

Again — step by step: In the function’s head we refer to Gradio’s Request object which allows us to read the cookies for this request. We then simply loop through all our config parameters, you may define as many as you want, according to your interface. If we find the particular cookie, we can set it’s value.

Finally we return the values. According to the outputs-argument from our demo.load( function (see above), we only return one value for one element.

4. Final Thoughts

If you want to see the process in action, check out my text-2-image playground at Hugging face either directly via https://huggingface.co/spaces/n42/pictero or using the easier-to-remember-domain: https://pictero.com (yes, you can use your own custom domains on Huggingface’s spaces — it’s a simply frame-redir in my DNS configuration).

I am not only using the aboved described workaround to persist users’s configuration: I also implemented a sharing feature reading the configuration from the URL GET-parameter config and put it into my interface.

Have fun and leave your thoughts!

Goal

You’ll learn how to leverage PyTorch, a machine learning library/framework, to construct a learning algorithm. Our focus will be on the cryptocurrency ADA, which operates on the Cardano blockchain.* Unlike most tutorials that solely consider price as the input feature for the model, we’ll incorporate volume and trade amounts. Implementing the sliding window method, we’ll introduce an ‘outlook gap,’ a less commonly utilized technique. Exploring various model architectures and optimizers, I’ll provide a rough explanation of their workings. Additionally, we’ll aim to enhance the model’s performance. Lastly, I’ll touch upon why a simple script and numerical data might not suffice for accurate price prediction: Stock, cryptos or butter. Whatever.

*)I’m not aiming to specifically promote cryptocurrencies. It’s simply convenient to access their historical prices. The primary focus is on PyTorch and machine learning.

Source Code

Jumping straight in without perusing my carefully crafted text? Bold move. But sure, go ahead, dive in. No hard feelings, probably. Check out this Colab:

https://drive.google.com/file/d/1lUef_DQBOV5Ngs5rLn3pGPWRpu9ho2Qd/view?usp=sharing

Let’s get some data

We’ll utilize the stock prices of ADA, a cryptocurrency operating on the Cardano blockchain. Kraken provides an extensive archive of historical data encompassing dozens of currencies across various time frames. I’ve opted for the 60-minute time resolution. Loading this data into a Pandas dataframe isn’t a significant challenge:

1df = pd.read_csv("data/ADAEUR_60.csv")
2df['date'] = pd.to_datetime(df['timestamp'], unit='s', errors='coerce')
3df['date'] = pd.to_datetime(df['date'])
4df.set_index('date', inplace=True)
5df

Visualize, Visualize, Visualize

One of the most crucial teachings from

Daniel Bourke

— a fantastic PyTorch and machine learning tutor, and likely much more. As a fellow enthusiast of visualizations, let’s kick things off by plotting the closing price and volume onto a single chart to gain insights into our data.”

 1# Downsample the data to not crash the plotting mechanism, we don't need to plot everything in the dataset
 2downsampled_df = df.resample('1D').mean()
 3# close price to the left y axis
 4plt.plot(downsampled_df.index, downsampled_df['close'], label='Close', color='blue')
 5plt.ylabel('Close', color='blue')
 6plt.tick_params(axis='y', labelcolor='blue')
 7# duplicate to get a second y axis on the right and plot the volune
 8ax2 = plt.twinx()
 9ax2.plot(downsampled_df.index, downsampled_df['volume'], label='Volume', color='red')
10ax2.set_ylabel('Volume', color='red')
11ax2.tick_params(axis='y', labelcolor='red')
12# Title and legend
13plt.title('Close Price vs. Volume')
14plt.show()py

I might not be a chart analyst, but here’s my take: Even if I had bought 1 million ADA in 2020 and sold them in 2021, I’d likely still be here, writing this article. Writing brings me joy.

Preparation

Let’s delve into the realm of writing machine learning code. I’ll begin by setting some hyperparameters. You might not grasp most of them yet, but we’ll cover them in detail later. For now, it’s beneficial to acknowledge their presence.

 1hidden_units = 64
 2num_layers = 4
 3learning_rate = 0.001
 4learning_rate_step_size=5
 5learning_rate_gamma=0.9
 6num_epochs = 100
 7batch_size = 32
 8window_size = 14
 9prediction_steps = 7
10dropout_rate = 0.2
11# stats and metrics
12epoch_count = []
13train_loss_values = []
14test_loss_values = []
15train_rmse_values = []
16test_rmse_values = []
17duration = 0
18features = ['close', 'volume', 'trades']
19target = 'close'

• Hidden Units: The quantity of internal memory cells within our model — visualize them as small algorithms aiding the learning process.
• Num_layers: Denotes the number of layers in the model, where each layer possesses its own memory cells.
• Learning Rate: Represents the magnitude by which the model adjusts its weights and biases (parameters) after each step.
• Learning Rate Step Size: Determines the frequency at which the learning rate should be incremented or decremented.
• Learning Rate Gamma: Serves as a multiplier to modulate the learning rate.
• Num_epochs: Signifies the number of iterations our primary loop will undergo during the training process.
• Batch Size: Specifies the quantity of data processed at a given instance.
• Window Size: Defines the number of past data points considered for our prediction.
• Prediction Steps: Indicates the number of data points to skip before reaching the data point we aim to predict.
• Dropout Rate: Determines the proportion of nodes in the model set to zero.

Lastly, we’ll designate the columns intended for our model input: close, volume, trades, and naturally, the column containing the price we aim to predict: close.

Data Normalisation

Another essential step involves splitting the data into training and test datasets. Initially, we normalize the data, a crucial process that not only accelerates computations but also enhances the overall quality of the model.

1# right now start with a small sample, as soon as we have enough computing power, we can skip this step
2df_sampled = df[features].head(1000).copy()
3#scaler = MinMaxScaler() # MinMax would work, too, but in fact a stock price has not really "min/max values", except the 0 ;)
4scaler = StandardScaler()
5# Extract the selected features and transform them
6selected_features = df_sampled[features].values.reshape(-1, len(features))
7scaled_features = scaler.fit_transform(selected_features)
8# Replace the original features with the scaled features in the DataFrame
9df_sampled[features] = scaled_features