published on 09 Oct, 2024. last updated on 05 Nov, 2024. approximately a 19 minute read.
as a part of wanting to claim my identity on the web, i started writing a static site generator in 2022. it's gone through several design phases but was ultimately shelved for a while. obviously, we're here now, with the closure of cohost finally spurring me to get it into a workable state. this is going to go into how it works, what it was intended to do, and where i still think it should go.
during the early days of the pandemic lockdown, when everyone was still taking it seriously, there was a sense in the air of community in online spaces. one of the projects that caught my eye around this time was the gemini protocol. in short, gemini wanted (wants?) to bridge the gap between the old internet and today, taking inspiration from both the web and from the gopher protocol. a mailing list refined ideas until a basic spec was formed, and it was designed in such a way as to resist future changes that would undermine its goals.
this project was really inspirational to me; they seemed to capture a lot of my philosophy with technology in a way i hadn't maybe been able to articulate up to then. first, a core design goal was that the protocol should be simple enough that it's feasible for a hobbyist to write a standards-compliant server or client in a weekend. i have a real love of home-grown software that i wrote to do exactly what i want and nothing else. there's something deeply satisfying about knowing exactly what's going on under the hood, and that if something goes wrong it's your fault and no one else's. this alone could have been enough to pique my interest.
but the gemini team also wanted to prevent the inevitability of ad tracking that rises with a technology such as the web, when it gets popular enough to commercialize. so it is strictly forbidden to download more than the file that the client requests. if you access a document over gemini, that single file is all that you will receive. this way, you always maintain control of who you request resources from, and therefore who you share your data with. this has the side effect of ensuring that the content of geminispace remains focused on just that: the content. if you take a dive into gemini, you'll find that the overwhelming majority of the content is text-focused blog posts.
so to author those text-focused documents, the natural question arose: what format to use? since a core goal was simplicity in implementation, HTML was out the window. there have been attempts to introduce a reduced subset of HTML, and indeed many voices have cried out that gemini as a protocol doesn't make any sense! why would you need a new protocol that nobody uses to serve documents when you can just make websites that behave according to your ideals? because it's too easy to stumble into websites that don't behave according to those ideals. gemini was proposed as a space where you knew that any gemini:// link you followed wouldn't violate those principles. whether that's worth it is an exercise left to the reader.
what about markdown? too many edge cases, too many specs, and technically it can allow HTML to be embedded anyway so that's a whole can of worms. i wasn't around for these conversations so i can't say what all was proposed, what reasons were given, but i know what they settled on: make a new document format. make it as dead simple to parse. we want homegrown clients, after all, how can we do that if they can't render the documents? and so gemtext was born.
gemtext, which actualizes as some weird mix of markdown and the gopher protocol's menu format, was the answer, and is the single most important thing to come out of this project for me. i've since gone through lists of markup formats to find anything that i like more, and i haven't been able to. at the core of gemtext is the notion that every line can be parsed on its own. there is no state change in the middle of a line, ever. you always know what kind of line you're working with based on the first three characters. this makes parsing gemtext so easy that i've written line-by-line parsers, AST (abstract syntax tree) converters, and pre-processor shims in several different languages with little effort. gemtext has been the most consistent underpinning of my site generator since its inception.
coming into this project, i don't think i really fully intended to write my own static site generator from nothing. my original idea was that i wanted to be able to write my posts once, and then publish them on both the gemini protocol and on the web side by side, and i wanted the experience to be exactly as simple and content-focused on both platforms. this made gemtext an easy decision to start with: converting it to HTML for display on the web could be done without a ton of effort, and converting it to gemtext for the gemini protocol was... obviously a non-issue.
i was going through something of a phase with the crystal programming language at the time, and alongside these goals, i had also fancied that i was going to write a gemini client and server as well. i don't really remember if it was for familiarity, nerd cred, or just a project that seemed like an approachable-yet-satisfying scope. so, hey, might as well write a central library in crystal that can parse gemtext, store it as an AST, and be able to convert it back out to any other format i want, right? this AST representation could also serve the clients for how to render the gemtext documents out to the screen. cool.
now if i want the presentation to be basically the same on gemini and the web, i'm going to need to style the webpage to look vaguely like a gemini client, right? i think i'd originally intended on using an existing static site generator, but once i had HTML files popping out of my converter, it was too easy to just start working on the CSS by hand. much of what i wrote in this initial phase is still present in the CSS as it exists today. and well, people don't want to click links to images on the web, they want the images embedded directly into the page! okay, i guess i'll need special handling for certain cases in my converter, then? these decisions are what ultimately turned this into a wholesale site generator unto itself.
and so i started writing my gemini client. it was a tui (terminal user interface), inspired by the lynx web browser. first, i was writing gemtext documents out to the console, then there was some ui work done, and then i was requesting documents over the gemini protocol, and i even had the pages scrolling, all just using the crystal standard library! wow, what a true realization of the gemini ethos. score one for me, right?
at the same time, i was toying with making the web version look how i want. i started with the principles of semantic HTML: there are a variety of tags, like <p> and <a> and <h1> that have a meaning when they're present in a document. i wanted to make sure that i retained those meanings. each regular text line in a gemtext document takes the role of a <p>, but browsers separate <p>s with blank lines by default, so i had to change that; i only want them to separate if there's a blank line between them in the gemtext document, because that's how they render in gemtext. i settled on a color scheme for the things i needed to style manually, and my desire was to respect the user's browser configurations as much as possible. default font faces and sizes, default background and foreground colors, that sort of thing. my dream of a website that the user could read exactly as they wanted was coming true, and i was the super genius at the center of this master plot, engineering a symphony of simplicity and freedom.
turns out people do things the way they do for a reason. my first reality check came in the form of respecting user fonts: most people don't configure their fonts, and for legacy reasons, the default fonts (sans, sans-serif, etc.) are atrocious to look at. i mean just really terrible to behold on modern systems. my page looked slick and modern in my firefox setup and like someone forgot a stylesheet on a site designed in 1998 for internet explorer almost anywhere else.
"no worries!" she thought, still not learning her lesson. "i'll use ui-sans-serif for my fonts! then at least they'll fit in with the modern system!" and indeed, this helped in safari. but, line spacing maybe isn't great by default in safari, and the default dark mode is white text on a black background in chrome. one at a time i kept adding new styles until the entire look of the site was bespoke. because the defaults are actively user hostile. for legacy reasons. maybe those gemini folks are really onto something.
i wouldn't know! i lost my crystal code for my gemini projects in an unfortunate laptop related disk mishap. me: professional web developer, git subject matter expert on my team, too lazy to bother with version control at home. haha, whoops.
enough of the website code stuck around from older versions i had on my file server, but i haven't gone back to the gemini client. it doesn't seem worth the effort any more. this was really the beginning of the end of my love affair with the protocol. i still love gemtext, and i'm still using it to this day to author these posts, but i almost never visit a capsule in geminispace any more. that's not to say i won't revisit it, but it just hasn't been a priority lately; the types of people i follow are on the web, they were on cohost, and now they're in my RSS reader. maybe gemini will gain traction with those folks, but for now it's relegated to the background for me. i'm not running a server for my website that i could stand up a gemini daemon on, i'm just using static hosting. i'm not a professional developer any more, i'm a sysadmin, and that means that maintaining and paying for an internet-facing server for a personal hobby project sounds about as fun as doing timesheets. i do that all day at work.
so, i put a lot of this down for a while. when i came back to dust it off, there was a lot of streamlining, polishing, and integrating to do. in something of a manic phase, i did a lot of that in the day or two leading up to my first post.
still here? we're getting nerdy now.
so, i do still have some crystal code from an attempted rewrite, based on creating a cli application to do conversions in the way pandoc or cmark does. i have the old code around from my original python prototype. i have an old attempt at the AST style conversion in ruby. i even have the code i used to format my cohost posts (rip), which relied on the same core conversion tool that i use now, with some bespoke elements for how cohost mangled posts. it let you use anchor links to directly link to a specific header within the cohost post, which would even work on the dashboard if the post was present. i was pretty chuffed with that.
yes, my cohost posts were written with this very static site generator i'm using now. i know, i'm so sexy.
so, what was turning into a lot of small shell scripts and a handful of ruby scripts has now been assembled into... one shell script with a lot of modes, and a handful of ruby scripts. the shell script is up to just over 100 lines, and has 11 functions. the ruby scripts, including embedded HTML templating, total under 400 lines at the time of writing; under 300 if you remove my buttons page, which we will get to soon, and which is almost entirely HTML, CSS, and javascript.
at the nucleus of this whole operation is gmi2html.rb, a sub-70 line script which iterates over a gemtext file and converts it line-by-line to HTML. this includes custom operations for my site: it adds embedded images with alt text in place of text links to images. it handles some custom links, where i can end a file with .ext and it'll replace it with .html, or prot:// will be replaced with https://. this is a carve out for absolute linking within my own site when i want the content mirrored onto gemini without any effort, down the line.
i've also added an escape hatch line type, which is a line starting with the ! character, where external scripts can be invoked and written in-line into the document at conversion time. this is how my published date and read time estimates at the top of my articles are made. in fact, this is how the first two lines of this document look:
# Technical Overview: Static Site Generator
! date readtime
if you're reading via RSS, you won't see those; we'll get to why in a moment.
the shell that sits around that nucleus is mkpage.rb, which contains HTML templating, title deduction from the first h1 element, the navigation sidebar (itself generated from a gemtext document). it also includes an invocation to gmitoc.rb if the page has enough headers to warrant a table of contents (RSS readers miss this too, if you're curious you can check out the full page). gmitoc.rb is a small script that generates the table from the headers in the document, outputting a gemtext formatted set of links that are intended to be rendered in monospace to look like a tree. this also makes a pass through the HTML converter. i've found gemtext to be an absolutely lovely intermediary format for things like this, honestly.
which brings us to mkindex.rb: a script with a small template for the main text of my index page, and then some code that outputs the list of links to all of my posts, in order, with date stamps. this is, of course, output in gemtext and then run through mkpage.rb to generate my index page. it gets the dates from a file that maintains published and updated dates for each post; more on that later.
the smallest ruby script is a custom CSS minimizer, weighing in at four actual lines of code of debatable readability. my intention with this was to reduce the weight of the file without sacrificing semantics, so it is essentially a whitespace-and-unnecessary-semicolon compactor. if you run the output through a code formatter, you should get exactly the same result as if you ran the input through it. this is enough to save 22% of the filesize of my current CSS file, and i've tested it on CSS as large as bootstrap just to make sure i wasn't making any false assumptions.
eugh, excuse the pun. before i could roll this site out, i wanted to make sure i had an atom feed for RSS readers, because there's been something of a revival in that space. in a way, this is the second output format of my multi-format approach, which would make gemini actually the third in line if i ever get to it.
the core of the feed generator isn't actually gmi2html.rb, but gmi2md.rb. this preprocesses the gemtext in much the same way: substituting in for prot:// and .ext, replacing links to images with inline images, and changing from gemtext's => link format to markdown's []. instead of allowing the ! escape hatch, it simply removes lines starting with !, as for now there are no reasons to include them in the feed. it then... just prints what it has, actually, because lists, preformatted text, and headers in gemtext are all valid markdown with the same meaning. the reason i chose to do this was that i didn't want to attempt to impose any styling on the feed content, but my html converter needs styling to look the way it's intended to be presented. markdown conversion will get me 90% of the way there without having to bother with maintaining my own running list of when i should be breaking up <p> elements on any basis more complicated than line-by-line. the end result isn't 1:1 with the gemini or web versions, but the content is 100% legible in the same way.
wrapped around this markdown conversion step is mkentry.rb, which takes as arguments the post path, the published and updated timestamps, and the site URL. it derives the title, runs gmi2md.rb into cmark (one of very few non-homegrown utilities in the stack), and populates an XML template that gets embedded into the feed. this gets called for each entry by mkfeed.rb, which essentially just iterates over the posts in the same manner that mkindex.rb does, but populating an atom feed XML template as it goes, rather than generating an intermediate step. almost all of these tools call out to my shell script to get titles, dates, and things like that; i tried to avoid duplicating those functions to the extent possible, so that if i change something in one place that change automatically cascades across the stack.
at this point, you may be following along and thinking
gosh, what a house of cards! are you invoking all those scripts by hand? how do you keep up with it all?
thank you for such a wonderful segue, my hypothetical friend, but before we get there we're going to take a quick detour to...
okay, it's a bit of a tease to talk about this page that i haven't published yet, but i want to get to it while we're still in ruby land. buttons.rb is a ruby script that contains a list of 88x31 images, their alt text, and the URLs they point to. this is such a cute little thing that some people have been doing that i was inspired to break my gemtext-only approach up to now. it outputs an HTML snippet with all the buttons, that i can eventually inline in another page using my ! escape hatch line type. the script is overwhelmingly HTML template with inline CSS and javascript, just getting populated from the list of images programatically.
ugh, javascript? i've been trying to avoid javascript up to now, but i decided it was worth its weight in evil for this particular use case. basically: i use 125% scaling on my laptop and my 4K monitor, which means that 88x31 renders on my screen as 110 by... 38.75 pixels. you could probably guess that the images lose their crispness... crisposity? the pixels get blurry, when this happens, even using the image-rendering: pixelated style. using media queries in CSS against the dppx value, which i believe is device pixels per px (the CSS unit px, not actual pixels, hence this problem in the first place; i hate web development), i've included custom styles that remedy this at 125%, 150%, 175%, and 200% scale.
but for whatever reason that just didn't seem like a good enough solution: what if someone has a scaling factor i hadn't considered? what if someone hates my text size and zooms to 80%? these people deserve maximum crisposity, too. so, i made a temporary alliance with my nemesis, javascript, because the CSS function that would resolve this for me hasn't been implemented yet.
so, basically, there's code on that page that listens for a change in the device pixel ratio, and generates a new media query that's exactly custom specified for your device's resolution. this means that no matter the zoom level or the display scaling, you should see a nice even integer interval of pixels when you view my buttons page. this post is coming out before the buttons page is, but trust me, it's cool. let me be proud in the abstract.
okay we're getting kind of lost in the atom metaphor here. i mean shell like shell scripting. it's a bash script. it's not like, the shell around a nucleus. in an atom. well, maybe it is. kind of? i probably should've thought this through first.
i have one bash script, util.sh, that does basically everything else that hasn't been explained up to now. its help output reads like this:
Usage:
add [post]
build [-f]
buttons
crush [size] [input] [output]
date [filename]
deploy
lastmod [filename]
new [title]
readtime [filename]
status
title [filename]
i can be really vague with the invocations because i'm the only person who uses it. this is truly all i need, to know what the hell is going on. this is one of the benefits of home-grown tools that you're the sole creator and only user of. i'll go through each one alphabetically here:
- add takes a [post] file as an argument, which is a .gmi document in the posts/ folder. it calculates the modified time using the date utility, in --rfc-3339=seconds mode, and updates my main posts.gmi file with the title, publish timestamp, and modified timestamp. it then runs mkpage.rb to automatically generate the HTML page and output it to my html/ output directory.
- build checks modified dates to see if any of the following steps are necessary, and executes them if so: outputting minimized CSS, regenerating the atom feed, and creating the index page. if the -f flag is passed, it forces all of these and also regenerates every post; this is useful for when i update the navigation sidebar, for instance.
- buttons just invokes buttons.rb. this is how the ! escape hatch gets the HTML that buttons.rb generates.
- crush takes a maximum resolution [size], an [input] file, and an [output] file. it uses imagemagick to scale the image down to the maximum, and then dither the image down to the "web safe ('netscape')" color palette. it also runs optipng to further reduce the size of the output file if applicable.
- date is a ! escape hatch utility that outputs the "published on day, year." text at the beginning of articles in my website. it derives this from my posts.gmi list, checking for [filename].
- deploy is an invocation to rclone that copies my html/ directory to my web host. this could just as easily be rsync or a git hook or whatever other utility one might use. the point is not to have to worry about it.
- lastmod is my ! escape hatch that does the same thing that 'date' does, but for modified time. if an article is updated in the future and i need to reflect that, i can just plop 'lastmod' on my ! line and be good to go.
- new takes a post [title], derives the filename for it, outputs the template (title header and ! line), and then opens the file in vim. this saves me from having to manually enter the same boilerplate each time.
- readtime is my ! escape hatch function that uses wc to approximate how long it might take to read [filename]. it's based on 250 words per minute and is calculated with awk.
- status checks my posts/ folder for untracked posts, and uses rclone to determine the differences between my local output directory and my hosting directory. this is just to see what's changed before a deploy, and could also just as easily be git based.
- title uses sed to check [filename] for the first instance of an h1, or #, and simply return the header line as a title. this gets called from a lot of different places.
and so that's how i interact with the site generator! new, add, build, deploy. updated a post? add, build, deploy. it's really come together to be far more usable than i ever intended or expected.
so, in the face of that development work, i have ended up with what you see before you. i want to stop and pay specific note to the 'crush' util function, because i didn't bother to explain that one yet.
i've always been a huge fan of the dithering aesthetic, and i thought that low-tech did something interesting with their site. they wanted you to be conscious of the fact that you're reading a page that's hosted on a solar-powered server. one of the ways they draw attention to this is with images that are downscaled, cross-hatch dithered, and colored with CSS to match the page. i wanted to do something similar with my site. it is explicitly not a bandwidth saving measure: a similar resolution jpg file will almost always look better and be smaller.
when i embed an image, i want you to be aware that it's serving to contextualize the text content. the focus is not on the fidelity of the image. it's crunchy, and it's got a weird spotty color quality to it, it's the opposite of new and shiny. i deliberately selected the "web-safe" color palette for this because i think it's a fun compromise of fidelity and cronch, but also because of the name. web-safe. that hearkens back to the days when you couldn't assume that a user's display could accurately depict much more than a couple hundred colors. i want you, when you engage with this page, to be thinking about how far computing has come. if you're browsing on a computer made in the past few years, i want you to think of what the web was like in the past.
i want you to think of what the web could still be like, if we wanted. i'm not telling you you need to make your images crunchy on purpose, but i think you should make your pages accessible. i use semantic HTML where i can, i respect the user's light or dark mode choices, and the entire site is usable with javascript disabled. it should be as readable on a modern smartphone, desktop firefox, dillo, or lynx. i haven't tested it, but i hope it's readable on the PSP's browser.
not everyone can access the latest devices with the fastest processors. not everyone has internet that's fast enough to download your gigantic images in the time it takes them to read the page. this website is an homage to anyone who can not, or chooses not to, accept the way most of the web works in 2024. i hope i'm doing a good job representing. i want to show that you can leverage modern web methodologies (CSS media queries, responsive design, javascript event listeners, whatever it may be) in a way that doesn't exclude older platforms, but progressively enhances the base that those older platforms built for you. i want to participate in a web that my eeepc wouldn't sweat over. there is, in my opinion, very little justification for much more than that.
hey, do you have accessibility tips for me? do you use a screen reader or have any other cool different ways of reading my posts? please reach out and let me know what i could do better!