-baseURL = "https://xangelo.ca"
-theme = "plain"
title = "Xangelo.ca"
author = "Angelo R"
copyright = "Copyright © 2011 - 2022"
paginate = 15
+baseURL = "https://xangelo.ca"
+theme = "brutalist-minimalist"
languageCode = "en-us"
+relativeURLs = true
enableInlineShortcodes = true
footnoteReturnLinkContents = "^"
rssLimit = 15
+[menu]
+[[menu.main]]
+name = 'About'
+pageRef = '/about'
+[[menu.main]]
+name = 'Linkblog'
+pageRef = '/linkblog'
+[[menu.main]]
+name = 'Categories'
+pageRef = '/categories'
+
[params]
subtitle = "Technical musings and other tidbits"
+siteBaseName = "Xangelo.ca"
+bloglistLength = 10
[markup]
[markup.highlight]
--- /dev/null
+---
+title: Friday, June 2nd, 2023
+tags: ["daily"]
+date: 2023-06-02T10:36:35.285-04:00
+---
+
+
+There was something magical about uploading the wordpress zip file to your shared host and going through `install.php`
+
+
+
+
+
+
+
\ No newline at end of file
--- /dev/null
+---
+title: Thursday, May 24, 2023
+tags: ["daily"]
+date: 2023-05-24T23:11:51.166-04:00
+---
+
+
+I'm attempting to adopt [Dave Winer's](https://scripting.com) posting setup of daily updates. I realize there's so much that I want to just say over the course of the day. Some of it might morph into full blown posts.. some of it will just disappear. I originally thought I would try and transition some into "tweets" and "toots" or whatever mastodon calls them.. but for now they'll just live here.
+
+I moved to hugo about a year ago, mostly to gain the easy github pages integration... I'm only now beginning to discover some of the neat details that made it so popular.
+
+
+
+
+
+
+
\ No newline at end of file
--- /dev/null
+---
+title: Thursday, May 25, 2023
+tags: ["daily"]
+date: 2023-05-25T21:20:53.011-04:00
+---
+
+
+I have a sneaking suspicion that [Azure DevOps](https://azure.microsoft.com/en-us/products/devops/) is way better than GitHub but no one is willing to admit it.
+
+For whatever reason this year has been really bad on my allergies. I'm trying a variety of over-the-counter medications but will likely have to resort to something a bit stronger before the summer is out.
+
+I just realized that all the "Daily Updates" are already one day out of sync due to the fact that I don't know what day it is. Just did an update to fix this once and for all.
+
+
+
+
+
+
+
\ No newline at end of file
--- /dev/null
+---
+title: Friday, May 26, 2023
+tags: ["daily"]
+date: 2023-05-26T13:33:16.642-04:00
+---
+
+
+Everyone wanting their own streaming service is just re-inventing cable and the whole reason that services like Netflix "distrupted". Gabe Newell said it best "[Piracy] is a service issue". And the service here sucks.
+
+
+
+
+
+
+
\ No newline at end of file
--- /dev/null
+---
+title: Tuesday, May 30, 2023
+tags: ["daily"]
+date: 2023-05-30T11:12:44.329-04:00
+---
+
+
+I feel like I'm ALMOST there on the process for deployments. I think right now I need to adjust the build process so that after I write a post in the outliner, it gets pushed up to github automatically (the raw data) which builds and deploys it to github pages. That way the "built" info isn't in github.
+
+
+
+
+
+
+
\ No newline at end of file
---
title: "About"
date: 2020-04-29T23:38:37-04:00
-draft: false
---
-Thanks for checking this out! It's been a number of years since I really changed by blog and this time I've decided to move to an all markdown blog since that's the way I do all my notes.
+I'm Angelo. I've been working in Software Development for the last 20 years or so, doing a bit of everything. I started my journey as a "Full Stack" developer/designer back when [2Advanced Studios launched v3](https://www.webdesignmuseum.org/exhibitions/2advanced-studios-v3-2001) and over the years I've moved further and further "backward" in the stack. Since about 2012 my technical focus has been around Backend/DevOps/SRE work and more broadly around "Leveling Up Engineering". I have a history of joining teams that are ready to take the next step from "We have this MVP" and helping them get to "We are a company". I've gotten to do a bit of everything from Founding my own Startup in the Security Space, Consulting (Both technical and Engineering Management), Leading teams as a People Manager, and contributing as a Principal Engineer. My own career seems to have a focus on Ed Tech + Security, with a bit of dabbling in Crypto Currency.
-Projects:
-- This site
-- Sketchy Heroes
-- BitBums
\ No newline at end of file
+This blog is a way for me to document all of the things I've learned and am still learning about everything I run across. The updates happen sporadically (as life does) and I hope that you'll be able to find something interesting here.
+
+If you have any questions, don't hesitate to drop me an email and we can figure it out.
--- /dev/null
+---
+title: Chrome HTTP Request stuck in "Stalled" State
+summary: Chrome reports that a request is stuck in a "stalled" state and I try and figure out why
+tags: ["chrome" ,"iis", "investigation", "explanation"]
+draft: true
+date: 2023-05-30T11:24:48.041-04:00
+---
+
+
+I got the chance to investigate a really odd bug where randomly network requests in Chrome would just hang. This would only occurr in our test environments at work and not in production. The request would hang for some long amount of time.. and then eventually complete successfully. The bug has been occurring for some time, but has been getting worse in Chrome. It got so bad that it was guaranteed that if you were using Chrome it was going to happen to you. Eventually it started happening in Firefox as well.. during an investor demo (what good is a demo if it doesn't go up in flames?). That's when I got roped in.
+
+The first thing I did was attempt to replicate it and capture a `.har` file that I could share with anyone else that was interested. This part was easy - just popping open the network tab, navigating to the app on a test env, and then clicking every link that would trigger a network request. After about 30-40 seconds I had replicated the event
+
+LINK TO TIMING WHERE IT SHOES THE QUEUE
+
+So we can clearly see here that the request took 2 minutes and the entirety of that time the connection was stuck in the `stalled` state. That indicates one of two things:
+
+1. Chrome never attempted to make the network request at all. Perhaps the priority on the request was dropped, maybe there were too many connections open to that FQDN already.
+
+1. In some situations chrome actually merges the CORS preflight requests into what it reports as `stalled`. So it's possible that there was a problem in the preflight request that caused the delay before the actual request happened.
+
+I did a bit more testing to figure out the scope of the issue. On/Off VPN with multiple browsers would all replicate the problem just fine. In this particular situation the fact that we could replicate it across EVERYTHING made it pretty clear that it was something related to our server.. but what? If the request never actually hits the server what could be going on?
+
+### Chrome Network Log
+
+One tool that chrome has to diagnose networking issues is hidden away at `chrome://net-export`. It generates a very VERY detailed log of everything network related that chrome is aware of.
+
+IMAGE OF CHROME CAPTURE NETWORK LOG
+
+I unchecked the `strip private information` option and told it to include cookies + credentials and started logging it to disk. Then I swapped back to my tab and replicated the issue. Waited a few seconds, and then went back and ended the captuer session.
+
+Once you get that capture file, you have to head over to https://netlog-viewer.appspot.com and import it. There's a TON of information here, and honestly I didn't even look at half of it. The only two things I cared about were the "Events" and "Timeline" sections. The Timeline really makes no sense until you have a idea of when your actual network event happened, so we can skip that and jump right over to Events
+
+There will likely be a lot of events. The "filter" at the top never worked for me given the sheer size of the events.. but scrolling through them all was just fine and eventually I found the URL request that caused the issue. If you click on the event it will display a bunch of debug information about the request.
+
+IMAGE OF EVENT 169281
+
+As you can see.. suddenly there's a HUGE jump in time from `66807` to `187631`. We've confirmed now that this is a problem that's occurring within the CORS preflight request specifically, and it's just getting rolled into the `stalled` state. The log viewer makes it trivial to dig down into the events and if you click on the details of the `HTTP_STREAM_JOB_CONTROLLER` event you can see some more details.
+
+IMAGE OF EVENT 169283
+
+Here again, we see that there is a definitely delay when it attempts to call `HTTP_STREAM_REQUEST_STARTED_JOB`
+
+IMAGE OF EVENT 169284
+
+And now we can easily see the problem: `SOCKET_POOL_STALLED_MAX_SOCKETS_PER_GROUP`
+
+For some reason, the socket pool dedicated to this particular group (in this case grouped by FQDN) is filled up and so it can't actually make the next request. So it just sits there.. until suddenly a socket is available (2 minutes later) and it is able to complete the rest of the request as expected.
+
+We can dig even further! Since we know that this is happening on an HTTP2 call, we can filter our events to only show us the http2 connections and that paints a more serious picture!
+
+Every since one of our http2 sockets is getting set a `GOAWAY` frame by the server.. but notice that it says `NO_ERROR`. Some quick research shows us that this is actually part of the "graceful shutdown" strategy. The server tells the client to `GOAWAY` but there's `NO_ERROR`. This is the clients indication to wrap up its work because the server is going to officially disconnect soon. Except that final disconnect frame is never sent. So as far as chrome is concerned, we're still happily connected so it returns the connection to the pool. But the server has disconnected.
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
---
title: "Code Reviews Are Failure"
date: 2022-02-24T11:05:34-05:00
-tags: ["git", "development"]
+tags: ["git", "development", "engineering processes"]
+summary: Code Reviews are nothing more than a half-hearted attempt to avoid planning
---
## Code Reviews are a Failure
title: "Removing the Default Font"
date: 2022-03-31T00:07:57-04:00
tags: ["upkeep", "fonts"]
+summary: Where I spend too long talking about why I removed the default font temporarily
---
This is a small change that I've made to the site that I've actually been thinking about for quite some time. I've always had a monospaced font configured in my CSS, forcing all text into whatever the default monospace font on your system is.
---
title: "DEVLOG: NewsRiver Updates"
date: 2020-11-10T09:43:04-05:00
+summary: A River-of-News style RSS Reader
---
## What is News River?
title: "Dungeon Generation in Roguelikes"
date: 2021-04-23T09:44:35-04:00
tags: ["roguelike", "explanation"]
+summary: A simple dungeon generation algorithm for organic rooms
---
---
## Introduction
-I've been working on Sketchy Heroes for a few years now
\ No newline at end of file
+I've been working on Sketchy Heroes for about 2 years now. Honestly it started as more of a learning
title: "Discovering Problems as a Manager"
date: 2021-11-01T17:11:57-04:00
tags: ["management"]
+draft: true
---
---
title: "Gemini, an HTTP alternative"
date: 2021-06-21T17:11:57-04:00
-tags: ["gemini"]
+tags: ["gemini", "upkeep"]
+summary: An HTTP alternative
---
For the last few months now I've started embracing my weird love for text and text-related user interfaces. From email to irc, I'm moving more and more of my social interactions away from traditional websites/apps and over to my terminal. A big part of this move is a step back from 3rd party applications to things that I have a bit more control over - but there are many reasons.
---
title: "Git Branching Strategies"
date: 2021-07-16T16:14:40-04:00
-tags: ["git", "development", "dev workflow"]
+tags: ["git", "development", "dev workflow", "engineering processes"]
+summary: A topic that every developer in the world has an opinion about
---
In a [previous](/posts/git-workflows/) post, I talked about git workflows and I outlined the two big problems I see with git workflows:
---
title: "Git Workflows Suck"
date: 2021-07-15T12:16:45-04:00
-tags: ["git", "development"]
+tags: ["git", "development", "engineering processes"]
+summary: Common pitfalls with everyones "git process"
---
I'm tired of all the git workflows. I'm tired of talking about the best way to craft commit messages. I'm tired of how people pile on the best way to do PRs. I'm sick of people picking apart architectural decisions in a PR. I'm tired of opening PRs only to have no one look at them for days while I continue to harass people into reviews. I'm over the fact that I don't think I've ever seen a single person clone a PR locally and actually verify that it does what it says it does.
title: "gitweb - a GitHub/GitLab alternative"
date: 2022-03-24T01:29:37-04:00
tags: ["git", "development", "dev tools", "gitweb"]
-draft: false
+summary: Setting up your own git repo browser
---
## Owning Your Digital Space
--- /dev/null
+---
+title: "My i3wm Setup"
+date: 2022-03-18T12:16:36-04:00
+draft: true
+---
+
+I've recently made the switch from the stock Ubuntu setup over to i3wm. I've
+used xmonad for many years before this, but always found it a bit harder than I
+would have liked to customize it.
+
+Most of what i3 does out of the box works perfectly for me. The only thing I
+didn't like was the fact that navigation was shifted from `hjkl` to `jkl;`. I'm
+also not a big fan of dmenu. Honestly I'm not totally sure why, but I've always
+struggled getting it to do what I wanted.
+
+With i3 I made a few small config changes, and swapped out the launcher and so
+far things have been quite excellent. I'm just posting this here so that I
+don't forget my configs.
+
+##
+
+
+## My Config
+As a vim user, I'm much more comfortable with the default vim bindings, than the "1 ket to the right" bindings that i3 tends to use.
--- /dev/null
+---
+title: "Mongo vs. DynamoDB"
+date: 2022-09-17T02:12:13-04:00
+draft: true
+tags: ["database", "mongo", "documentdb"]
+---
+
+## Abstract
+Whenever you get into the conversation of NoSQL, and specifically "Document based" systems, you inevitably seem to end up on either DynamoDB or Mongo. The truth is while DynamoDB is a wonderful tool, Mongo should probably be your first stop.
+
+## Mongo vs. DynamoDB
+Mongo and DynamoDB are two document storage databases in the NoSQL family. Where mongo took the more traditional "server" route, DynamoDB opted for "serverless". As a result, most of the distinction between the two comes from this divide.
+### The infrastructure and Scaling
+### The UX
+### Support
---
title: "Designing On-Call"
date: 2022-09-21T22:36:23-04:00
-tags: ["engineering", "engineering org"]
+tags: ["engineering", "engineering org", "engineering processes"]
+summary: How to implement a successful on-call process for the first time
---
On-call is one of those things that all developers end up doing at some point. My goal isn't to discuss the merit of on-call, but rather what the point of on-call is and how to go about designing what “on-call” means at your company. I'm going to start at the very beginning because chances are you're already doing it wrong. I should also note that I'm looking at this specifically from a SaaS point of view.
--- /dev/null
+---
+title: Now Powered by Outlines
+tags: ["outliner", "blog", "project"]
+date: 2023-05-21T23:29:45.732-04:00
+summary: An update to how I publish new posts
+---
+
+
+One of the things that I do every so often is completely re-write the backend of my blog. I've mostly hit upon a UI that I like, but I've swapped out the backend over the years between various custom iterations, wordpress, ghost, and now finally Hugo. This time, I've swapped out how I write my blog posts - but kept everything else the same.
+
+The current system allows me to write markdown in vim. I'm normally running `hugo serve -w` at the same time so I can watch the rendered version of what I'm doing as I go. It's sort of like a hacked-together live preview. It works well enough.
+
+However, for the last 10 years (maybe more?) I've been a huge fan of outliners. I original started with various projects by [Dave Winer](https://scripting.com) and I used almost everything he's written around them for a number of years. I've also tried tooling like [Workflowy](https://workflowy.com) and almost every other infinite-bullet-list tool that came after them. They were all.. fine? I had no real problems with them except that they never really stuck around for very long. They were in a tab in my browser, and my browser has like 100 tabs open at any given time.
+
+For the last 6 months or so, however, I've been working on my own outliner. It started as an in-browser tool... and I quickly moved it to an offline-first desktop app via [Tauri](https://tauri.app). Having it offline first meant a few big things.
+
+1. All the syncing tools just work. Dropbox, rsync, backblaze, s3 as a filesystem. Whatever. It works. All nodes in the outliner are individual json files on your computer. There's an "outline" file that stitches the nodes together into a tree. <br>
+
+2. Building desktop first allowed me to bypass the need for user accounts and passwords. It allowed me to skip out on the complexities of providing reliable encrypted storage to users. I don't need to run a collection of servers and databases and object stores to power this thing.
+
+Since I use this tool across operating systems all day long it slowly ended up being where most of my thoughts for blog ideas end up. And so what I hope to be the final feature of this project was realized: I'd love to be able to write my whole blog post in here, and then hit a button to have it publish to my hugo deploy.
+
+This first iteration uses a lot of hard-coded stuff.. and I'll probably take some time to iron out some of the edge cases around rendering.. but it honestly came together pretty quickly. Since every node in the outliner is markdown it was trivial to put it together. As of right now, I can write my blog post in my outliner, press `shift+p` and have it write out a markdown file to my local hugo instance.
+
+For now, I do some manual reviewing before officially publishing it. For now there's a few more usability things that need to be added like
+
+- differentiating which posts are published vs. un-published
+
+- being able to 'unpublish' a node
+
+But honestly? I'm kind of enjoying this right now.
+
+
+
+
+
+
+
--- /dev/null
+---
+title: "Running a Git Server"
+date: 2022-03-30T11:08:53-04:00
+draft: true
+---
+
+# Running your own Git server
+Following this theme of owning your digital space I wanted to do a quick post on what it takes to run your own git server. There are a bunch of services out there like that will host your git repositories for free (I personally recommend [sourcehut](https://sourcehut.org/)), but you can get a most of what they offer for free with [gitweb](https://xangelo.ca/posts/gitweb/).
+
+## Why would you want to do this?
+Honestly it's not as complicated as you would think to run your own git server and get almost all the same functionality from popular forges. Running your own git server doesn't mean you can't use something like GitHub or GitLab - it's not a one or the other choice. You can host all your repos on your own VPS while still pushing to GitHub/GitLab to ensure that you can still take advantage of their social networks.
+
+For me the biggest precursor to hosting my own repos was just to get more familiar with git. I've been using git for a number of years, but it was only in the last year or so that I really started digging into all the functionality that it offers.
+
+In addition to that there's a huge difference in complexity between running your own server, and running a service for millions of others to consume. As GitHub grows (both in size and in feature set) the likelihood of any one piece of it being down also grows. If we look at the [Incident History](https://www.githubstatus.com/history) of GitHub from Jan 2022 to March 2022 we can see that some piece of it has been dozens of incidents and outages. Using git allows you to bypass a lot of those problems, but centralizing on any hosted forge will always come with these inherent risks. It's much better to use hosted forges as secondary locations and just host your own git repos.
+
+## What do you need?
+1. Server - You need somewhere to host your git server. Personally I use [Digital Ocean]() but you are free to use anything that gives you ssh access and an IP address.
+1. You'll need root access to that server
+
+That's it. Those are the minimum requirements to actually host your git repos elsewhere. Everything else on top of that is just extra functionality and we'll talk about those later.
+
+### A Server
+The easiest thing to do would be to spin up a brand new server on a cloud platform like [DigitalOcean](https://digitalocean.com). It costs approx 5$ CAD and is probably one of the cheaper shared hosts you can find. You don't need anything fancy because the git repos actually do nothing the majority of the time. There's only activity on them when you push/pull.
+
+If you spin up a droplet (server) in Digital Ocean it will provide you the ability to set up your root credentials. Personally I'd suggest using an SSH key! It's a heck of a lot easier than typing in your password every time you try and connect.
+
+#### ssh-keygen
--- /dev/null
+---
+title: Simple Redis Job Queue
+summary: A naive redis job queue implementation
+tags: ["redis", "architecture"]
+date: 2023-05-25T00:15:53.362-04:00
+---
+
+
+A common pattern in most software projects is the "queue". You'll throw some stuff on there and eventually get around to sorting it all out. In some cases, you may not even really care about that.
+
+I recent ran into a situation in node.js where I needed to offload some CPU intensive out of the main http request handling process. Originally I thought about just forking the process, but I due to the fact that this endpoint is public facing (albeit authenticated and paid for), I was wary. Instead I figured I'd rope in a quick job queue.
+
+### Caveats:
+
+- The jobs have an easy "deduplication key". An ID that is the same given the same inputs.<br>
+
+- If we don't complete a job for any reason, that's fine, another one will likely show up again in a few minutes
+
+We were also using redis so here's a real simple pattern that can handle this kind of workload.
+
+1. Whenever we get a new job we calculate its deduplication ID. We'll call this the `jobId` moving forward.
+
+1. We call `set job:[jobId] value NX EX 10`. This sets the key `job:[jobId]` to some value we provide only if the key does not already exist. It also sets a 10s expiration time. We're using `NX` to provide some additional deduplication and we're using `EX` to ensure that if for some reason our job doesn't complete in 10s.. just allow another job to be inserted with the same key. You can tune `EX` as necessary, or leave it off entirely if you never want this job re-processed again until you flush your cache.
+
+1. We use `rpush [queueName] [jobId]` to add the job Id to the list of jobs that need to be processed.
+
+1. If we have any additional information about the job we want to pass on, we can use `hset jobDetails:[jobId] k1 v1 k2 v2...` and store it in there.
+
+Your "worker" is just a process that runs `lrange [queueName] 0 0`. That will retrieve the oldest `jobId` in your queue. You can grab any further information from the `jobDetails:[jobId]` hash set and do whatever work you need. When you're done you can call `ltrim [queueName] 0 0` which will remove the job from the queue.
+
+An interesting fact about this setup is that all your calls are `O(1)` and you can pipeline the initial `set`/`rpush`/`hset` calls so that things are even faster.
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
-Subproject commit eea052235660a87a2684fd485fe17b2de43e3e00
+Subproject commit 3882c078f7c356a5adcfeaeb58fb293385431092
--- /dev/null
+Subproject commit 739f8ff2d55f0922c2480f1674928ae30f2844bb
Home / xangelo.ca.git / commitdiff