• On a Friday evening in June, our customer’s RPA platform (UIPath) that writes their staff’s schedule back into their source system failed. It was reported after hours — and the team that owns it doesn’t monitor it on weekends or after hours. So it sat dark, unattended, all weekend.
• It never reported a single error. 592 tasks over five and a half days, zero failures logged. The bot accepted every job and picked up none of them. Inbound traffic looked perfectly normal. Their dashboards were green.
• We knew it broke right away, because a separate service we’d built months earlier doesn’t ask the bot whether the bot succeeded — it re-imports the schedule from the source system and checks what’s actually true. Our monitoring system lit up on Friday night while their dashboards were green.
• Over that weekend, with Claude and Codex as pair programmers, I built Scribe — a service that performs the same write-backs UIPath did. First code was in production at 2:33 AM Monday. A build that would normally take weeks took a weekend.
• We never turned it off. Five weeks later, with UIPath fully healthy, 17.7% of its writes still need Scribe to fix them. The outage was just the loud version of a problem we’d been fixing manually ourselves.

The dependency we didn’t control

A bit of context on what we do. Our platform provides an AI scheduling assistant that manages healthcare staffing schedules — swaps, coverage, pickups, partial shifts, schedule changes. When a change is finalized in our system, it has to be written back into the customer’s source scheduling system, the system of record.

The vendor for that source system is building out a proper API, but it isn’t ready yet. So in the meantime, the write-back happens the way a human would do it: by driving the scheduling UI. And because of our customer’s requirements, those writes don’t run on our infrastructure — they’re routed to their RPA team, who own and operate a fleet of UIPath bots. We hand them the work; their bots type it in.

That’s a clean separation of duties on paper. In practice it means a critical step in our pipeline runs on a platform we can see but cannot fix — operated by a team on a different schedule than ours.

The Friday night nobody was watching

On Friday, June 5, the RPA system failed. The report came in after hours. And here’s the detail that matters most: the team that owns that platform doesn’t monitor it on weekends or after hours.

So the failure just… sat there. From Friday evening until Monday morning, the thing responsible for writing our users’ schedule changes into their system of record was completely dark, and the only people who could fix it weren’t looking. The root cause was on the UIPath side — the same class of infrastructure failure described in this writeup — and it would take their IT team a week to get the fix in place.

How we knew: the confirmation service

Months before Scribe existed, we’d built something unglamorous: a confirmation service that independently verifies the round trip.

The logic is simple. Our system finalizes a schedule change. UIPath writes it into the source system. Then, separately, we re-import the schedule from the source system on an ongoing basis. The confirmation service checks that every finalized request actually shows up in that re-import. If a change doesn’t materialize within a threshold, it alerts — faster when the affected shift starts soon, because an unwritten shift starting in two hours is a genuine staffing emergency, not a data quality nit.

Crucially, it doesn’t trust the writer’s own report of success. It doesn’t ask UIPath whether UIPath succeeded. It asks the system of record what’s actually true, and compares. That independence is exactly why it worked when the bot was dead: a failed write and a lying write look identical to that service, and both trip the alarm.

As our monitoring service flagged issues, we would route them to the original RPA team to beef up their bot and manually fix the source system ourselves. The team took turns clearing daily alerts. Sometimes we’d have one alert and sometimes a few but it always seemed manageable.

So on a Friday night, with nobody at the RPA team watching, our confirmation alerts told us the truth — nothing was landing. Not some things. Nothing. It wasn’t the 1-2 issues we’d normally see, it was pouring in. That’s how we knew UIPath was completely down.

The number that still bothers me

I went back and queried production while writing this, and the result is worth sitting with.

Across the five and a half days UIPath was down, 592 RPA tasks (aka scheduling changes) were created. Exactly zero of them failed.

Not a single error. Not a single abandoned. The bot never threw. It never reported a problem. The system was hard down — the jobs weren’t attempted and botched, they were accepted into the queue, moved to NEW, and then never picked up at all. If you had been watching UIPath’s own status reporting that weekend, you would have seen a clean board.

You can watch it flatline in the completion timestamps:

Normal throughput was ~135 a day. The last completion of the pre-outage era landed Friday afternoon, and then the queue simply stopped draining. Not slowly. Not with errors. It just stopped.

There was no error to catch and no exception to page on. The only observable difference between “working perfectly” and “completely dead” was whether the work actually appeared in the source system — which is to say, the only thing that could tell the difference was a system that went and looked.

This is the thing I’d put on a poster: the failure was invisible in every system that reported on itself, and obvious in the one system that checked reality. A silent failure and a successful write look identical from the inside. They look completely different from the outside, if you bother to look from the outside.

That service is the unsung hero here. Everything that follows was possible because we already had ground truth.

The weekend

I had a choice. Wait an unknown number of days for someone else’s unmonitored outage to get noticed and fixed, or build the write-back myself.

We already had the hard-won knowledge of how to drive the source system’s UI — that’s the exact knowledge UIPath encoded. What we didn’t have was a service to run it. So over the weekend of June 6–7, I sat down with Claude and Codex and built one. (Why both: that’s a story for another blog post.)

The work was not trivial: browser automation against a finicky, hydration-heavy scheduling UI; a task-routing layer; a verification loop; a full session lifecycle with retries, alerting, and stuck-session recovery. On my own that’s a multi-week project — most of it the slow, unglamorous work of handling the twentieth weird edge case in a web UI that wasn’t built to be automated.

With AI in the loop, the ratio inverted. I spent my time deciding what the system should do and reviewing whether each piece was correct; the models handled the mechanical breadth — the selectors, the retry scans, the confirmation logic, the tests. Scribe’s first code was running in production at 2:33 AM Monday — the headless-browser worker image, then the add-shift automation, then the post-create confirmation logic. By Monday afternoon the full verify-and-remediate service was deployed and we were processing messages on behalf of the RPA team.

What made the weekend possible wasn’t just fast code generation — it was a fast loop. Every commit deployed to production within minutes, so I wasn’t writing against a mental model of how the source system’s UI behaved; I was writing against how it actually behaved, right now, with real schedule data flowing through. Find an edge case, fix it, ship it, watch the next session, repeat. Over the following days that loop produced dozens of small, surgical hardening deploys — late-hydrating grid cells, clobbered time fields, detached iframes, locked schedules — each live minutes after I understood the problem. And by I, I mean a looped skill that reviewed recent Scribe Sessions, built a fix, automated a test through the training environment and then opened a PR. That cadence, with AI absorbing the mechanical cost of each fix, is what let a weekend build survive contact with production.

I want to be precise, because “AI built it in a weekend” is usually exaggerated. I still made every architectural decision. But the calendar doesn’t lie: a build that would have eaten weeks took a weekend, and it went into production handling real schedule writes during a real outage.

Giving it back (and taking it right back)

On June 10, the customer’s team thought UIPath was ready. We flipped the bypass off and handed the writes back.

It still wasn’t right. We hit issues immediately and flipped the bypass back on — Scribe resumed, and we kept going. It wasn’t until June 11 that UIPath genuinely took back over.

That failed handoff is why the next part exists. Recovery isn’t a moment, it’s a negotiation — and I no longer wanted our users’ schedules depending on someone else’s declaration that things were fine.

Meanwhile, the roadmap didn’t stop

Here’s the detail that makes me laugh now and did not make me laugh then.

We had a major feature launching that same week: a new Nurse Residency program, in development since early May. It pre-schedules an entire year of classes, up front, for every incoming resident across every cohort. It is exactly the kind of launch you want to happen on a calm week.

It went out on Thursday, June 11 at 3:09 PM — about six hours after UIPath had finally, genuinely taken back over that morning. In one shot it created 3,096 class shifts across 10 cohorts, which became 3,085 shift-adds, which became 3,922 RPA write-back tasks.

For scale: the entire outage — five and a half days of a dead platform — produced 592 tasks. The residency launch produced almost seven times that in an afternoon.

And it went through Scribe. 3,704 of those write-backs were routed to the bot we’d finished building 72 hours earlier, on a system whose first line of code had existed for three days. That’s the day I stopped thinking of Scribe as an emergency measure. An emergency measure gets you through the emergency; it doesn’t absorb your biggest feature change that month three days after it was written.

I’d love to claim we planned it that way. We didn’t. The launch date and the outage had nothing to do with each other; they just collided. But that’s sort of the point: you don’t get to schedule the week when your dependencies fail, and you don’t get to pause your roadmap while they sort it out.

Belts, suspenders, and another pair of suspenders

When UIPath came back for real, the obvious move was to switch Scribe off. I didn’t. The outage had made the actual risk legible: a single write-back, on any platform, operated by a team that doesn’t watch it after hours, is a single point of failure with a multi-day blast radius.

So we kept Scribe running and added layers. Today the stack looks like this:

1. Independent detection (predates Scribe). The confirmation service verifies every finalized change against the schedule re-imported from the source system. It never trusts the writer’s self-report — it checks the system of record. This is ground truth, and it’s the backstop under everything else.

2. Automatic takeover when the bot stalls. If a job sits in UIPath’s queue in the NEW stage for more than 20 minutes, we delete it from their queue and take it over ourselves. This is the direct answer to “nobody monitors it after hours.” An unattended failure on their side can no longer quietly become a problem for our users on ours — a stalled job now has a deadline, and when it blows it, we just do the work.

3. Scribe as stand-in. When UIPath is unavailable wholesale, a routing switch sends everything to Scribe. That’s what carried us through the outage week, and it’s still there.

4. Scribe as botsitter. In normal operation UIPath does the primary write, and Scribe independently re-reads the live schedule afterward, diffs it against what our system intended, and remediates discrepancies on the spot. UIPath does the work; Scribe checks the work and fixes what’s wrong — immediately, without waiting for a human to notice.

5. The eventual API. The vendor’s real API, when it lands, becomes the primary path — and every layer above stays exactly where it is.

Read those together and the shape is: we detect independently, we take over automatically, we verify everything a bot claims it did, and we confirm the round trip from the source of truth regardless of who wrote it. Any single layer failing no longer means schedule drift for a nurse who needs to know when they’re working.

Was keeping it on worth it?

Our botsitter keeps running now that UIPath is stable. Take a representative week, five weeks after the outage ended. Scribe ran 395 sessions. 389 of them executed, and 345 of those were adjudicating an actual UIPath write — the other 44 were confirmations of schedule imports from elsewhere, which Scribe checks but UIPath never touched. Of the 345 that were grading UIPath’s homework:

• 284 (82.3%) came back verified — UIPath did the work correctly, Scribe confirmed it, nothing to do.
• 61 (17.7%) came back remediated — Scribe found a discrepancy between what we intended and what the source system actually showed, and fixed it on the spot.

More than one in every six automated writes still needs correcting. Not during a crisis — during a totally normal week, on a healthy platform. Those aren’t outages; they’re the quiet stuff. An edit that silently didn’t commit. A shift that didn’t have a note written in explaining what changed. Each one is a nurse whose schedule says something different from what they agreed to, and none of them would have thrown an error. Previously we would have manually fixed these ourselves, but now we have a system that can catch these issues automatically and fix them in real-time.

The stuck-job takeover fired 7 times that week, too. Seven jobs that sat in the queue past 20 minutes and would otherwise have waited for someone to notice.

That’s my answer on whether to turn it off. The outage was never the real problem — it was just the loud version of a quiet one we’d been living with.

The outage was the forcing function. The defense-in-depth is what we kept.

The technical deep-dive

For the engineers. Here’s roughly how it’s built.

Scribe drives the real UI, in Python. It uses Playwright (headless Chromium, baked into our worker image) to log into the source scheduling system and perform the same actions a human scheduler would: navigate to a date, open a unit grid, add a shift, edit a shift’s time or notes, read a row back to confirm. There’s no secret API — it’s the same surface UIPath automated, which is exactly why Scribe could stand in one-to-one.

Everything is a verify-and-remediate session. The core object is a ScribeSession with an explicit lifecycle: pending → processing → remediation_planned → remediating → remediated, plus terminal states like needs_review and remediation_failed. A session spawns per parent request (a swap, a coverage, a schedule change) once that parent’s sibling tasks have settled. It re-scans the live grid, diffs against intended state, and either confirms a match or plans concrete adjustments.

“Settled” includes “stuck.” This is the takeover layer in code. A task settles normally when it reaches a terminal status (COMPLETE/ERROR/ABANDONED/BYPASSED). But a task sitting in NEW — accepted into UIPath’s queue and never picked up — blocks only until a staleness threshold passes (SCRIBE_STUCK_NEW_THRESHOLD_MINUTES, currently 20). Past that, we cancel the queue item and dispatch a session with trigger_reason=STUCK_NEW. A bot that never starts is now just another settled state, and the pipeline keeps moving.

Routing is a single switch. An RPA_BYPASS_UIPATH path resubmits a parent’s tasks so they flow to Scribe instead of the UIPath queue, canceling any live queue item first. During the outage that switch was “everything routes to Scribe.” It’s the scriptable, auditable equivalent of an admin “resubmit these tasks” action — which is what let us flip back on June 10, discover UIPath still wasn’t right, and flip off again within the hour without a deploy.

The confirmation service is deliberately separate. It lives in a different app (schedule_requests), predates Scribe by months, and shares no code path with the writers. It reconciles finalized requests against the re-imported schedule, with escalating thresholds — a standard alert window, a tighter one for shifts starting within hours — and an ImportHealth gate so ordinary import lag doesn’t fire false alarms. Keeping it independent is the whole point: if it shared plumbing with the thing it’s checking, it would fail in the same ways at the same time.

The hard part was the web UI, not the logic. A freshly-changed grid hydrates asynchronously — a new cell can render a beat after the loader clears — so a naive scan misses it and wrongly concludes the write failed. Scribe does bounded re-scans before deciding a cell is absent. Add-confirmation is date-pinned to the target day-column to avoid opening every same-time panel on a busy unit. Saves are confirmed by toast and by re-reading the new row’s id. Edits set the time field last, after the panel hydrates, because setting it early gets clobbered. Transient iframe detaches self-heal. Every one of those lines exists because of a specific way the UI lied to us — and that long tail is exactly the work AI made survivable in a weekend.

Concurrency is guarded. A database constraint enforces at most one open session per parent request. Stale workers that try to update a session already reaped or replaced hit a Superseded guard and back off instead of racing. Status transitions are filtered updates (update where status = expected), so two workers can’t both advance the same session. Sessions run on a dedicated Celery worker so a wedged browser can’t starve the rest of the queue.

It fails loud, not silent. A scheduled reaper sweeps sessions stuck past a threshold — re-dispatching stale pending rows, retrying stale active ones below the attempt cap, hard-failing the rest. An attention monitor emails our internal team — deliberately not the customer’s RPA address — whenever a session needs human eyes. The whole point of a botsitter is that it never fails quietly. A discrepancy it can’t fix becomes an alert, not a surprise a week later.

The takeaway

The headline is that Claude and Codex let one engineer ship a production RPA replacement over a weekend. That’s true, and it’s remarkable.

But the lesson I keep coming back to is quieter, and airplanes got there first. A 777’s flight controls don’t just carry three computers — they carry three channels, each running three lanes on different processors, compiled by different compilers, isolated physically and electrically. The redundancy isn’t the count; it’s the dissimilarity. Three identical computers don’t vote, they agree — confidently, simultaneously, right up until they’re all wrong together.

That’s what the confirmation service is, and why it’s the least exciting code we own. It isn’t a third opinion about whether the write landed — it’s the only thing in the room that isn’t an opinion. UIPath’s report and Scribe’s report both rest on the same assumption: that the system doing the work can tell you whether the work happened. The confirmation service doesn’t ask. It goes and looks. When every voter can be wrong the same way, the tie-breaker can’t be another vote — it has to be the one that reads reality instead of reporting on itself. It’s the reason we weren’t blind while the people responsible for the failure were afk.

AI gave us the speed to replace a dependency in 48 hours. But speed only helps if you already know something is broken. Build the thing that tells you the truth first.

It’s funny that I’m finding time to write in the middle of a hackathon at 48-in-48. After joining three teams, it seems I’ve completed everything on my list and I’m waiting on Stephanie to wrap up. So that must mean it’s a great time to start writing again and create a blog entry! Hopefully I can write up something about this experience when I’m done.

Back to static pages for your web application.

The benefit of splitting out your main app from your static marketing pages is creating an enviornment the best suits your authors/editors, designers and engineers. Take Clockwise.MD for instance, we have a well defined SDLC process for making changes to the web application. It includes testing, peer review, approvals and carefully selecting which memoral quote, interesting fact or animated GIF to add to your PR approval. The flip side is our head of marketing shouldn’t need an extensive PR process to add a new pet to our Team page.

Similar to the earlier article I mentioned, we needed to load static pages up for our app but we don’t want to host those files on our actual web server. Instead, we created our web site using Jekyll. The resulting static site is uploaded to S3 with a CDN in front to make the site load fast. Then any files we don’t want to serve up from our web application, we proxy down from our CDN and send it to the requestor. We also cache it so that it’s super fast.

class PagesController < ApplicationController
  def www_base
    expires_in 1.hour, public: true
    response = s3_pages(request.path)
    render status: response.code, inline: response.body
  end

  private

  def s3_pages(path)
    uri = URI('https://your-s3-or-cdn-site.com' + adjusted_path(path))
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.ciphers = 'TLSv1.2:!aNULL:!eNULL'
    http.ssl_version = :TLSv1_2
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER
    request = Net::HTTP::Get.new(uri.request_uri)
    http.request(request)
  end
end

Then at the bottom of your routes.rb file, add:

  # external pages
  root to: 'pages#www_base'
  get '*unmatched_route', to: 'pages#www_base'

Now your home page will hit pages#www_base. Additionally, any page that’s not already specified earlier on your web application’s routes will be routed to your static site. If a user hits /products, then that same request will get served up from your static site. If a user hits /foo/bar and that path doesn’t exist, then they’ll even get your static site’s 404 page and www_base will send back the correct 404 status code as well.

To make things easier for your marketing folks, let them edit pages using Github’s online editors then setup a CI process to rebuild and redeploy the static site on every commit. Instant updates without having to the install the Wordpress malware.

There are many suggestions as to how to fix this, so I tried a few.

• Sharing Host Volumes with Docker Containers is a good overview of multiple things to try by Oliver Günther.
• I started with docker-osx-dev but received rsync errors after running it for a few minutes.
• Switched to VMWare Fusion and that didn’t help so maybe it’s not VirtualBox file sharing.
• Tried docker-rysnc under Fusion but it gave me:

rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(453) [sender=2.6.9]
error: exit status 12
Watching for file changes ...

Maybe this magic bullet shit won’t work, let’s try using an image as part of compose so that we’re more explicit about this syncing process.

• Looks like docker-unison works to sync files and it syncs them quickly but the application still takes 20 seconds to load a simple page.
• dinghy created a VM as a wrapper to docker-machine and no luck with that too.

Other NFS scripting options:

• Boot2docker: Using nfs instead of vboxsf to mount /Users
• How Blackfire leverages Docker under the File Sharing section
• boot2docker-nfs.rb is a simple gist working with VirtualBox

Since I have Fusion installed though, this seems like a lot of work to just get a basic Rails app to run.

My solution:

• Use Docker for everything not Rails or Guard. Postgres and workers seem fast enough.
• Run Rails locally with your standard gem install bundler; bundle.
• Leave the app running in Docker Compose if I need to check out anything in Linux but avoid it at all costs.

I’ve got too much real work to do.

Required Steps

My first stop was to follow Thoughtbot’s The Hitchhiker’s Guide to Riding a Mountain Lion.

• Get Xcode and Command Line Tools installed
• Fix homebrew permissions with sudo chown -R $USER /usr/local
• Update homebrew with brew update
• Install gcc 4.2 to install older rubies with brew tap homebrew/dupes; brew install autoconf automake apple-gcc42
• Install X11 support with XQuartz

TL;DR for installing Ruby 1.8.7

$ export CPPFLAGS=-I/opt/X11/include
$ CC=/usr/local/bin/gcc-4.2 rvm install 1.8.7

My little story

Initially I didn’t want to install X11 since I didn’t think I needed it.

Then I tried to use vim (not MacVim, I just wanted to edit a file quickly) and received:

$ vim
Vim: Caught deadly signal SEGV
Vim: Finished.
[1]    42573 segmentation fault  vim

What version of vim?

$ vim --version
VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Jun 20 2012 13:16:02)
Compiled by root@apple.com
/snip/

It seems that if you brew install vim, it still breaks:

$ /usr/local/bin/vim --version
VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Oct 15 2012 18:02:59)
MacOS X (unix) version
Included patches: 1-687
Compiled by databyte@nyx

Turns out that I need ruby-1.8.7 due to a compatibility in Janus, which was outdated. The quick fix was upgrading Janus by running rake within ~/.vim but I might as well compile ruby-1.8.7 since I bothered to install gcc 4.2.

Let’s try to install ruby-1.8.7 with rvm:

$ rvm install ruby-1.8.7-p370
No binary rubies available for: osx/10.8/x86_64/ruby-1.8.7-p370.
Continuing with compilation. Please read 'rvm mount' to get more information on binary rubies.
Installing Ruby from source to: /Users/databyte/.rvm/rubies/ruby-1.8.7-p370, this may take a while depending on your cpu(s)...
ruby-1.8.7-p370 - #downloading ruby-1.8.7-p370, this may take a while depending on your connection...
ruby-1.8.7-p370 - #extracting ruby-1.8.7-p370 to /Users/databyte/.rvm/src/ruby-1.8.7-p370
ruby-1.8.7-p370 - #extracted to /Users/databyte/.rvm/src/ruby-1.8.7-p370
Applying patch /Users/databyte/.rvm/patches/ruby/1.8.7/stdout-rouge-fix.patch
Applying patch /Users/databyte/.rvm/patches/ruby/1.8.7/no_sslv2.diff
ruby-1.8.7-p370 - #configuring
ruby-1.8.7-p370 - #compiling
Error running 'make', please read /Users/databyte/.rvm/log/ruby-1.8.7-p370/make.log
There has been an error while running make. Halting the installation.
Please be aware that you just installed a ruby that requires        2 patches just to be compiled on up to date linux system.
This may have known and unaccounted for security vulnerabilities.
Please consider upgrading to Ruby 1.9.3-286 which will have all of the latest security patches.

Turns out in the logs that:

/usr/include/tk.h:78:23: error: X11/Xlib.h: No such file or directory

Hence the need for XQuartz. Now let’s install XQuartz and then reinstall ruby but also explicity set X11’s include path:

$ export CPPFLAGS=-I/opt/X11/include
$ CC=/usr/local/bin/gcc-4.2 rvm reinstall 1.8.7
Removing /Users/databyte/.rvm/src/ruby-1.8.7-p370...
/Users/databyte/.rvm/rubies/ruby-1.8.7-p370 has already been removed.
No binary rubies available for: osx/10.8/x86_64/ruby-1.8.7-p370.
Continuing with compilation. Please read 'rvm mount' to get more information on binary rubies.
Installing Ruby from source to: /Users/databyte/.rvm/rubies/ruby-1.8.7-p370, this may take a while depending on your cpu(s)...
ruby-1.8.7-p370 - #downloading ruby-1.8.7-p370, this may take a while depending on your connection...
ruby-1.8.7-p370 - #extracting ruby-1.8.7-p370 to /Users/databyte/.rvm/src/ruby-1.8.7-p370
ruby-1.8.7-p370 - #extracted to /Users/databyte/.rvm/src/ruby-1.8.7-p370
Applying patch /Users/databyte/.rvm/patches/ruby/1.8.7/stdout-rouge-fix.patch
Applying patch /Users/databyte/.rvm/patches/ruby/1.8.7/no_sslv2.diff
ruby-1.8.7-p370 - #configuring
ruby-1.8.7-p370 - #compiling
ruby-1.8.7-p370 - #installing
Removing old Rubygems files...
Installing rubygems-1.8.24 for ruby-1.8.7-p370 ...
Installation of rubygems completed successfully.
Saving wrappers to '/Users/databyte/.rvm/bin'.
ruby-1.8.7-p370 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
ruby-1.8.7-p370 - #importing default gemsets (/Users/databyte/.rvm/gemsets/)
Install of ruby-1.8.7-p370 - #complete
Please be aware that you just installed a ruby that requires        2 patches just to be compiled on up to date linux system.
This may have known and unaccounted for security vulnerabilities.
Please consider upgrading to Ruby 1.9.3-286 which will have all of the latest security patches.
Making gemset ruby-1.8.7-p370 pristine.
Making gemset ruby-1.8.7-p370@global pristine.

Well that works and now I have ruby 1.8.7. Let’s try out vim again:

$ rvm use 1.8.7
$ /usr/local/bin/vim

Yep, that works too!

$ vim --version
VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Oct 15 2012 18:08:38)
MacOS X (unix) version
Included patches: 1-687
Compiled by databyte@
Huge version without GUI.  Features included (+) or not (-):
+arabic +autocmd -balloon_eval -browse ++builtin_terms +byte_offset +cindent
-clientserver +clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
+conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con +diff +digraphs
-dnd -ebcdic +emacs_tags +eval +ex_extra +extra_search +farsi +file_in_path
+find_in_path +float +folding -footer +fork() -gettext -hangul_input +iconv
+insert_expand +jumplist +keymap +langmap +libcall +linebreak +lispindent
+listcmds +localmap -lua +menu +mksession +modify_fname +mouse -mouseshape
+mouse_dec -mouse_gpm -mouse_jsbterm +mouse_netterm -mouse_sysmouse
+mouse_xterm +mouse_urxvt +mouse_sgr +multi_byte +multi_lang -mzscheme
+netbeans_intg +path_extra -perl +persistent_undo +postscript +printer +profile
 +python -python3 +quickfix +reltime +rightleft +ruby +scrollbind +signs
+smartindent -sniff +startuptime +statusline -sun_workshop +syntax +tag_binary
+tag_old_static -tag_any_white -tcl +terminfo +termresponse +textobjects +title
 -toolbar +user_commands +vertsplit +virtualedit +visual +visualextra +viminfo
+vreplace +wildignore +wildmenu +windows +writebackup -X11 -xfontset -xim -xsmp
 -xterm_clipboard -xterm_save
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: cc -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X_UNIX -no-cpp-precomp  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: cc   -L.    -L/usr/local/lib -o vim       -lm  -lncurses -liconv -framework Cocoa     -framework Python   -lruby -lobjc

In New York, I’ve attended and helped many developers in the local NYC on Rails meetups through a weekly meeting called Hacker Hours. It’s a nice change of pace from the normal Ruby meetups because Hacker Hours attracts a lot of non-Ruby developers in a wide range of skill ranges over a wider diverse set of people. I helped in Ruby, PHP, .NET, Java, HTML/CSS/JavaScript, iPhone, etc.

I have plans in motion to migrate back to Atlanta and once I settle in, I plan to create the Help Club.

The rules are simple:

1. You do talk about Help Club.
2. You DO tweet about Help Club.
3. You need to show up to learn or to help.

Based on rule #3, the meeting will lack suites and recruiters. This is for engineers looking to accelerate their learning or mentoring skills with complete comfort.

The process of helping or being helped is encouraged and shared through a ranking process. For every activity you participate in, you will earn points.

• 1 pt per meeting when learning.
• 3 pts per meeting when teaching.
• 10 pts for a lightning talk.
• 25 pts for a short presentation.

Those points go towards a rank.

• 10 pts White Belt
• 50 pts Yellow Belt
• 100 pts Green Belt
• 250 pts Brown Belt
• 500 pts Black Belt

I’ll be the idiot grinning and cheerleading everyone as I hand out belts.

My plan is to have a bi-weekly 2.5 hour meeting. We will have two 1 hour helping sessions with a 30 minute presentation block in the middle. The presentations will either be a single short presentation or three lightning talks.

I look forward to working on this idea with my future coworkers and in encouraging everyone to learn. From beginners to experts. From desktop applications to mobile and web sites. I’ll even bring in Arduino and Raspberry Pi devices if there are interested developers.

Suggestions welcomed and I hope to announce something soon!

Status page a few days ago:

Now, they’re combining multiple months together to decrease the number.

Status page today:

The numbers are looking better and better eh?

Three questions:

1. Why are they listing July separate (which was a good month right?) and then the 3 previous months with July included again?
2. What happened to January through April?
3. Is this going to be a rolling previous three months? Or is next month going to drop May and June then list only July and August?

I wish their marketing department didn’t make these pages.

They should list percentages for the current month and the previous month with a link to a page listing all the months ever recorded.

Even better would be to mimic your parent company (Salesforce) or Amazon. The way they list history is much easier to digest.

Amazon:

Salesforce:

I, for one, couldn’t care less if you used Ruby or Rails, Python or Django, C# or ASP.NET. I do care that you would a) only stick with what you know and b) choose PHP.

For the former, it’s simple - you should assume you simply don’t know everything there is to know and continually try to better yourself. If you haven’t reached that conclusion yet, the book The Passionate Programmer will surely help.

For the latter, hmm, PHP eh?

I’ve done PHP development as recently as last year but to read into more of a PHP’ers point of view, I recently read a blog post named PHP is much better than what you think.

In it, the author, Fabien Potencier, sounds just like the person I mentioned earlier. The opening paragraphs calls PHP the easiest, most popular and quickest language. All great reasons to pick one language over another. So let’s break this down.

The Language

PHP is old but instead of a language that moves forward with its syntax, PHP chooses to build on top of the old cruft and never clears away the cobwebs. Its syntax coherence is below most other languages. A blog titled PHP: a fractal of bad design says it better than anyone else:

• PHP is full of surprises: mysql_real_escape_string, E_ALL
• PHP is inconsistent: strpos, str_rot13
• PHP requires boilerplate: error-checking around C API calls, ===
• PHP is flaky: ==, foreach ($foo as &$bar)
• PHP is opaque: no stack traces by default or for fatals, complex error reporting

Consistency is the key to discovering a new language, framework or API. One of the key arguments for PHP is that it’s easy. I don’t see how inconsistency is easy. From the same article:

• Underscore versus not: strpos/str_rot13, php_uname/phpversion, base64_encode/urlencode, gettype/get_class
• “to” versus 2: ascii2ebcdic, bin2hex, deg2rad, strtolower, strtotime
• Object+verb versus verb+object: base64_decode, str_shuffle, var_dump versus create_function, recode_string
• Argument order: array_filter($input, $callback) versus array_map($callback, $input), strpos($haystack, $needle) versus array_search($needle, $haystack)
• Prefix confusion: usleep versus microtime
• Case insensitive functions vary on where the i goes in the name.
• About half the array functions actually start with array_. The others do not.
• htmlentities and html_entity_decode are inverses of each other, with completely different naming conventions.

Seriously read through that entire article, it’s a great read.

The Ecosystem

The author mentions git but git is everywhere. Even Microsoft uses. it. now.

It’s great to see PHP has a dependency manager but so does everyone else. The fact that you have one and it works decently does not mean it’s better than another language or implementation. At this point, not having a dependency manager is a problem. All modern frameworks have them including RubyGems, Python PyPI, Node Package Manager (NPM), Microsoft NuGet, etc.

I’m not sure how the author judges that Composer is better than other package managers. Is it because it resolves dependencies or has over 2400 packages? I don’t know. The other managers do handle dependencies as well and the package counts as of today are:

• Composer at 2,454
• RubyGems at 42,278
• PyPI at 22,860
• NuGet at 7,792

Under the original article, the author calls this community collaboration but really he’s making a point about the number of packages available in PHP. Given the numbers, I expected a lot more from PHP given the number of sites online and the years its been in service.

The Community

To me, the community is not about sharing code but more about the people themselves. The developers make the community because they strive to make everything better. They don’t sit in their own echo chambers and pat each other on the back, talking about how great this is about their preferred language or that is about their favorite framework. They see the shortcomings and they strive to address them while acknowledging their strengths and weaknesses.

The main issue I see with PHP but also with some other communities is the lack of sharing. Not just code but technical advise, support and overall camaraderie. Some communities are getting better at it. Having spent years in the Microsoft and Java ecosystems, I’ve seen it shape up better in Microsoft but that’s because they have evangelists running around doing their job. Java’s “ok” but Oracle hasn’t helped at all, especially given recent lawsuits and Hudson vs Jenkins.

The JavaScript community is an interesting one and the developers within it range from copy/paste scripters to those that thrive on functional programming who have dived deep into CoffeeScript or Dart to know which OOP parts suck compared to, say, LiveScript.

That wide variety of developers makes it harder to polarize the community as a whole and makes it difficult for one great programmer to recognize another. It’s similar to PHP.

Conclusion

The language is not easy to learn if it’s inconsistent. The community around your language and framework really matters; it allows you to learn and grow to become a passionate programmer.

Ultimately, I suggest that all developers should learn more than one language and framework. Get out your bubble and expand your horizon.

You can then choose the right language and the right platform for the right problem. Just as long as it’s not PHP.

A developer I helped out there recently learned Rails and joined a new startup. He asked me an interesting question regarding working at the new startup:

The other dev is very productive. I feel very slow and unproductive. Is this what it’s like every time you join a project? Everything is just a friggin’ mess?

Ignorance or Incapable?

I think there’s a distinction between ignorance, which is not knowing something because you haven’t had the opportunity, and being incapable, meaning not being able to do it even though you know how. When I start new projects, it takes hours and days of reading to get past the ignorance stage. It takes even longer if they use external libraries or APIs that I’ve not used before.

Don’t expect to be productive within days or even weeks. In larger projects, it could take months.

Legacy Code

On some projects, the developers are disorganized and unskilled or even advanced developers who used so much meta programming that the code reverted itself into being even harder to read. I have to read through the mess and the confusion to determine the program’s purpose and the programmer’s intent.

A developer’s goal is to make the program meet the goals of the business and it’s just as important to make it readable to other developers.

I think we would all agree that Ruby and Python reads easier than Assembler though all languages are capable of meeting their business needs. So you have to look at your highly productive developer and ask, is she writing functioning code, readable code or both?

Falling behind

From the new developer:

I’m burning out because I stare at “magic” all day and produce at a snail’s pace. I don’t get the reward of finishing something frequently and it decreases my morale.

If pairing through large chunks of the application isn’t an option, I suggest focusing on something small that’s part of the whole and working on it until you grok it. You can also start on a new section of code based on technology you already know.

Starting on a new project with lots of existing code and developers can be a daunting proposition. Your first commit will probably be a bug fix or small tweak.

Personally, I always start in the tests. That’s an easy place to find the interesting business logic and the uninteresting CRUD code. Test code needs to be refactored and structured just as much as the code it tests which provides another opportunity to learn more about the codebase and remove more of that “magic”.

Doing FAST vs Doing it RIGHT

Make your code a shining example of how it should work. Don’t spend too much time polishing it. Any true craftsman will want to continue working on their craft and as they get better and learn more, you’ll look back at your old code and wonder what you were thinking.

Any code written more than an hour ago is crap and will eventually be revisited. One of the Ruby Rogues said that all code is experimental.

Therefore, take the time to do it right. Don’t try to play catch-up and throw away quality in favor of speed.

The difference between that highly productive developer that writes messy code and the code that you write starts to shift the culture from one of productive mess which inherently contains technical debt to quality code that’s easier to work with later.

It’s always better to know about the engineering culture before you sign the dotted line but if it’s truly a startup, then you can shape its future.

Readable code

Recently there was a popular gist from a contractor that was let go due to applying aggressive SRP to a simple PostComment class in Ruby. The thread is a good read but when I think of the new developer’s comments regarding “magic” code, I think of something a previous co-worker, Donald Ball, wrote:

… getting code to do what it’s supposed to do is the lowest threshold for acceptability in my book. Getting code to clearly communicate its intent and to be easy and safe to modify is a higher bar, but is the standard by which I judge any code that’s intended to work for longer than a day. We spend more time reading code than writing code.