gram: page

> ./2025-09-04.rebuild.lab.md

Lenses
(coming soon!)

---
name: Lab Rebuild / Summer 2025
summary: Do I need an excuse for such a long absence? I demolished my lab; such a bad idea.
composer: c4lliope
labels:
  - lab
  - deploy
---

For me, August 2025 is going to remain
the most spectacularly unplanned and chaotic mess of garbled memories,
where each day carried so much nuance,
and so many challenges,
that there has been scarce occasion to pause and breathe,
to reshape the experiences into more than the layered percussions
of sun, moon, sea, sail, spoke, and signal.

I hope there is a chance to re-use that opening paragraph,
in a closer examination of the days of August,
only I need to begin by describing changes happening across this domain.

If you happen to be reading this,
or perhaps if your RSS reader is doing a refresh,
before 6pm on Thurs 09-04,
you may only see a partial essay.

- - -

## Background: Realizing a long-held dream.

I had good reason, cause, and chance to move my app deployment,
from the basement where I lease a static IP address,
to my local laptop, which I've had to leave up and running all night,
in a hacker-space called the [Baltimore Node](baltimorenode.org).

I'm sure others have likely managed before me,
a production online app running from a laptop.
In my case I am especially pleased to do so,
as a reinforcement of the aims described
in the business' [incorporación proposal].

When I composed the proposal in 2024 I was inexperienced in essays,
and I need to apologize to the DC employees
who had to read through that rubbish;
moreso because as I began this company I was also in a phase
of psychedelic experimentación.

If you dig through that essay,
in a sense a legally binding one,
you'll see the key theme is an idea I call "sublimación".

I realize now that my founding essay
was simply a poor re-hashing of an idea
explained much more clearly and simply
in the [Local-First Essay] by Ink & Switch.
They open by saying:

> Cloud apps like Google Docs and Trello are popular
because they enable real-time collaboration with colleagues,
and they make it easy for us to access our work from all of our devices.
However, by centralizing data storage on servers,
cloud apps also take away ownership and agency from users.
If a service shuts down, the software stops functioning,
and data created with that software is lost.

It has been said before in the "local-first space"
that numerous people are reaching the same conclusion
from different initial conditions.
The realización that our building processes are flawed,
doomed to someday fail, is a process that can occur in a dispersed manner,
like the many [origins of infintessimal calculus],
or the rich explosion of humanist philosophies during the [Axial Age]

However the conclusion is reached, here we are.

Each of hundreds of programmers and small businesses
are choosing a small niche in the local-first space.
[A full index] is a good place to begin for end-users
who are sick of usual app-store norms,
and [LoFi.so] has many resources to choose from,
including the [Local-First Discord][discord].
Need to read more? See [Local-First News][news].

The area I chose as a primary concern is unusual for the space,
which is likely why I had been such a late-comer,
missing the forest for the trees.

I chose to focus on web-app deployments.

[incorporación proposal]: /chronicle/corp
[Local-First Essay]: https://www.inkandswitch.com/essay/local-first/
[origins of infinitesimal calculus]: https://en.wikipedia.org/wiki/History_of_calculus
[Axial Age]: https://en.wikipedia.org/wiki/Axial_Age
[A full index]: https://www.localfirstsoftware.com/
[LoFi.so]: https://lofi.so/
[discord]: https://discord.gg/ZRrwZxn4rW
[news]: https://www.localfirstnews.com/2025-09-04/

- - -

## 2025 Aim: Reliable Deploys

Of course, 'web-app' can be seen as an antithesis to 'local-first'.
If you agree, maybe I can change your mind.

Yesterday I came across an app I'd like to build into the mix,
called [Shiori]. The purpose is to aggregate and label bookmarks,
and enable an offline-reader mode for easy mobile access.
There is no cloud-based company managing the program,
and the features drop it categorically into the local-first space.

Shiori remains a web-app,
and is capable enough for many users to access,
although the primary audience is for a single-user per deployment.
To run Shiori, users are encouraged to launch a web domain themselves.

Many apps are being produced in this manner
because of the rising popularity of home-labbing,
perhaps known to some as self-hosting.
Besides, the web has made up some really amazing norms
for apps that are dynamic and accessible,
and is much more pleasing to build for than most "native" OS apps.

To launch your app's domain,
the normal approach is to lease a machine from a cloud company.
In this case, for such a simple app, the easiest option may be [PikaPods].
More experienced coders are familiar - and correspondingly sick of -
flexible options like [Linode] and [DigitalOcean],
which probably are happy to lend our login credentials
to the nearest NSA field office at the mere whisper of enforcement.

So, until now every domain manager had two choices:

- add to Amazon AWS or Akamai's global reach,
by signing on to use their machines
(usually repackaged by a cloud middle-manager)

- order a business internet plan,
and consent to days-on-end
of random phone calls with your ISP,
to order a static IP address rigged up.

I'm partial to the second option,
because reaching local-first dominance
has to both amplify the share of local apps and programs,
and crucially,
**reduce our dependence on cloud data centers**.

I'd like to see a report each billing cycle,
describing how much water my apps drank
while they were being cooled by Google's
electron-warehouse plumbing.

The deplorable shape of our industry is plain to see
when passing through Ashburn, VA -
a close suburb for me to reach,
where there is no business, no one in sight,
and only a bland shopping mall for amusement.

These are human prices of our industry
that need to be recognized and addressed.

[Shiori]: https://github.com/go-shiori/shiori
[PikaPods]: https://www.pikapods.com/apps
[Linode]: https://www.linode.com/
[DigitalOcean]: https://www.digitalocean.com/

- - -

## March 2025: Baseline Lab

Although my company's purpose is to bring the web back to the home,
a further complicación had been... my lack of a home.

I am nomadic,
carrying my home around in a pickup truck or a sailboat each day,
and so I'd like to accomplish
as many of my aims as possible
while being as locación-independent as possible.

For the static IP address, I called up my pals in Upper Marlboro, MD;
there is a home there full of the most generous people I know.
I arranged for a business internet line to their home,
and placed a couple machines in their basement.

I'd been commuting from place to place,
and would occasionally find a chance to go by and check on the machines,
manage upgrades, plug and unplug cords.
The biggest flub had been encrypting the boot drive;
each reboot required hands on a physical keybard.
Nonetheless, the process remained in place through much of August.
Six months on a static IP address!

Then... let's discuss how August went.

- - -

## (beginning) August 2025: Sailing Fiasco

The "friend's basement" playbook is, eh,
more or less manageable when you live in close proximity,
are capable of a deep bond of co-reliance,
and are equal shareholders in the success of the program.

I'm unsure how I scored on any one of those marks,
though the deployment held up until
a small crisis landed on Aug 1,
requiring me to immediately set sail.

A sailboat is an ideal place to run cyber exercises.
Because there is no escape from the physical isolación,
the only connección you are likely to find to other humans
has to be relayed to you through radio signals.

I dove deep into the [Self-Hosted Podcast][sh.show]
and [Linux Unplugged][unplugged],
and called into many [cancer research calls][compbio],
where the group was far more eager (or enabled)
for me to put the "comp" into "computational biology"
than they were to put it into "compensation".

Of course, I had experiments to run.
I launched [Mumble].
I launched [Traccar].
I launched [n8n], and shoe-horned comment fields into the essay pages.

And I broke the lab. Again. And again.

To the dear residents of <u>Larry Cat House</u>,

> I am so so sorry, sincerely and immensely so,
for the endless calls on Signal
where I explained complex passcodes and cable-plugging procedures,
and the "quick sanity checks" where none was found.

I had to balance downtime for a [core-business client app][pain],
my need for experimentación as a coping mechanism, for lonely breezeless days,
and the superb annoyance I caused my pals back on land each day,
immediately after promising it (probably?) wouldn't happen again.

I hung on like this until I got back to land,
and spent one day fixing up the lab
to get the blog running again before heading to [HOPE].
I promised myself I would do no experiments while I was in NYC,
and besides I had no more energy to do so.

I was getting enough of a cyber-social buzz from the conference,
and NYC is the deepest end of the social spectrum to drown in,
after two weeks on a rolling bouncing bay.

After NYC, I headed back to the lab.
I had ideas to manage, more [grand plans][plan].
I had decided to separate the apps and the experiments,
leave the production app running as I brought the bulk of the lab
to Baltimore Node, where I could exhaust days on end
running necessary experiments and building at high speed.

With all but the core production machine "baseboard" loaded into my truck,
I locked the door to my friend's house behind me
and began to pull out of their neighborhood.

Then I remembered. I should check if the app is online!

Not only no app, but no signal. No SSH.

I spun around, rang the doorbell, met the new roommate,
and pulled the final cord out of Baseline Lab.

[sh.show]: https://selfhosted.show/
[unplugged]: https://linuxunplugged.com/
[compbio]: https://compbio.dmvpetridish.com/
[Mumble]: https://www.mumble.info/
[Traccar]: https://www.traccar.org/
[n8n]: https://n8n.io/
[pain]: https://book.painawayofphilly.com
[HOPE]: https://hope.net/
[plan]: /chronicle/publishing.plan

- - -

## (end) August 2025: (re)Building Blocks

However much sleep I had lost in the preceding days,
I was soon to lose more.
My usually inattentive client quickly messaged me,
saying "the app is down. Can we host it somewhere else?"
and of course I said "yes, please".

I recommended [fly.io] because heck,
I'm finally beginning to recognize my mission
as a smidge on the idealistic side.
If at all possible I can keep my ideals from holding up my pals' business,
I suppose I should.

I burned a couple days building a proper Elixir/Phoenix `Dockerfile`,
double-checking port and IP bindings between my `config/*.exs` files,
and I **finally** had a reliable deployment that no longer breaks the memory limits.

On my breaks, I honed my process for launching NixOS on a Raspberry Pi,
and soon headed back to Larry Cat House to plug one in; then a second.

The first pi I launched is called `pebble`,
and for around a week this had been running the blog.
On a couple occasions, I checked in, and thought I saw
that the pi had crashed; then, a few hours or the next day,
the app would be online again.

Clearly, the app is seeing more than zero traffic,
and the pi is unable to keep up.
No problem; the second pi I plugged in
enabled me to experiment with [Nebula].
Nebula, in essence, enables me to spread the lab out
to any number of geographically dispersed areas.

I placed some Nebula certificates on the two lab Pi's,
assigned certs for my laptop (`chesapeake`),
and with only a few days
of config & network-address munging behind me,
I can finally keep a reliable connection up between my
`chesapeake`, in Baltimore, and `pebble`, in Upper Marlboro.

[fly.io]: https://fly.io/
[Nebula]: https://github.com/slackhq/nebula

- - -

## September 2025: Dispersed

As much as I like the reliable connection between here and there,
the glory lies in the opposite direction:
I can keep up a reliable connection from **there to here**.

Remember, `there`, where `pebble` is running,
is the business line static IP address.
`pebble` is only a low-grade Raspberry Pi 4B with 2GB RAM,
purchased before I knew what to look for in a single-board computer.
`pebble` kept choking and lagging
when asked to compile markdown for a blogging engine.

When `pebble` can reach me here, on `chesapeake`,
a full-blown x86 coding machine, the game fully changes.
I kept the blog running at full capacity,
by disabling my idle timer through the night,
leaving the laptop plugged in to charge,
on a residential wifi connection,
reaching far higher speeds than the pi could manage.

I kept the blog running even as I ran Blender,
and downloaded a bunch of YouTube videos explaining how to use Blender.
I kept the blog running even as I **compiled Blender from scratch**,
because that's how things go on NixOS,
and there was no single hiccup in the response or lag from the app.

Finally, I logged on this morning and began
playing around with the essay,
and realized that as I saved my progress,
the rendered page displayed the changes -
on the public `://operand.online` domain.

In essence, I signed up my pals for a static IP address,
and made sure my domain's DNS records all point to that address.
Then, I rigged up Nebula and Caddy to reference an address
on a Nebula virtual-LAN; the magical, ephemeral, dispersed layer
that replaces all networking headaches with certificate-management headaches.

As long as Nebula and Caddy agree on an IP address for the app machine,
and Caddy and the app-machine agree on a port number for the app,
and Caddy and the app agree on a domain name,
then the app can be run on any machine, any place,
regardless of network setup.

Of course, this experimentation process
has required me to make serious changes
to the [business plan][plan],
so that much of what was described under `BASE`
is now marked as 'broken' in red,
and moved to `NODE`,
where I can be more hands-on with my experiments,
for much less risk.

- - -

## A Moment, to Breathe

Some may say I am in a precarious posición, with my business.
Why be nomadic, why go sailing, why be so un-employ-<s>ed</s> -able?
There are few role models to choose from, when breaking so many norms,
so I am lucky to rely on [essays that stretch the seams on this world.][threadings]

In her essays Ismatu examines the social scenarios of her tribe,
and the consequences of being raised a child of two nacións.

I feel an immense bond with this composer,
who I hope I shall be lucky enough to speak to,
and collaborate alongside someday,
by my own happenstance of being born in London,
raised in Tokio, by midwestern parents.

Global children like us
are fully incapable of considering the nonsense
of a flat-earth hypotheses,
because we have seen the curvature of the earth
from the window seats of numerous long cross-oceanic flights.

Global children like us are less prone to ensnarement
by a corporate mission statement,i spun around, rang the doorbell, met the new roommate, and pulled the final cord out of baseline lab.
when we recognize that the mission ends where the state's boundary lies,
less prone to being hooked by a viral ad campaign
when we can examine a global virus campaigning through places once called home.

I'm sure that in the middle of the largest rapidly globalizing crisis
our species has yet encountered,
all one can hope to do is find an area where their skills can shine,
press them into full gear,
and share the consequences publicly as loud and openly as possible.

In my case, this consequence seems to be the simultaneous dissolución
of the core mechanic underlying the business models
of Amazon AWS, Google Cloud, and Microsoft Azure;
the mechanism laid in place by ISP providers such as Comcast, AT&T, and Verizon,
many years ago when the first phone lines began being repurposed for internet purposes.

The static IP has been the bane of my career,
and in the end this windmill has been blown away by the deployment
of a single Raspberry Pi, running a background Nebula process.

[threadings]: http://threadings.io/

- - -

## Going ahead: a Reproducible Base

Again, nearly nobody I know has a static IP address,
and more and more people I know have apps they'd like to launch,
and spare old machines lying around home with no purpose,
and gradually depreciating innards. This is the problem of sublimación;
the economic resource of silicon doomed to disuse,
by lack of access to the demanded use cases.

By chance, Baseline Lab is my second failed static-IP experiment;
the first one being in my parents' place of business in Michigan.
Seeing as I have two static IPs to make use of,
a business model comes to mind.

Soon, you'll be able to log in to `://operand.online`,
and for a choose-your-donación price,
or pay-by-bandwidth for larger apps,
you can send along your domain name to us,
and I'll issue you a Nebula certificate.
You can choose a static IP address from our list of nodes,
to use as a relay for your web app.
Any one should do, so choose based on geography.

You can assign your DNS records to our static IP,
and our Caddy reverse proxy can begin handling your app's
call-and-response through our Nebula mesh.

For memberships needing more nodes on a single mesh,
perhaps you'd be open to on-demand cert allocación & revocación
from a secure app, on a dedicated nebula mesh.

Of course,
no one can build on such a core open-source library such as Nebula,
unless you also run open-source.
No problems here, this has been the case for Operand since day one.

I can see many docs pages ahead,
for gradually-sharpened essay skills to go to use.