Once a Maintainer: Mike Dalessio
The nokogiri maintainer on how AI has changed security and on maintainer burnout
Welcome to Once a Maintainer, where we interview open source maintainers and talk about the work they do.
This week we’re talking to Mike Dalessio, longtime Ruby maintainer (nokogiri, loofah, mechanize to name a few) and member of the Rails Committer team. Mike works on security, infrastructure, and performance at 37signals.
Once a Maintainer is written by the team at Infield, a platform for managing open source software upgrades.
I’d love to get started by hearing how you got into software development. What was your first exposure to writing software?
When I was a kid, I had a friend who lived up the street whose dad bought him a VIC-20, which was like the Commodore 64 predecessor. And we used to play text-only adventure games on it, and then we started to write our own. That was kind of when I started realizing oh, you can tell these things what to do and they’re fun. But then, you know, years went by and I got into college where I was a physics major. And so much of modern physics is compute intensive and computer intensive that I ended up having to learn how to program in order to be useful. I had a gig as a research assistant for like, $5 an hour or something, processing Voyager data. And I needed to learn this computer language. And I just got locked in a room for days on end where I needed to figure out how SunOS worked and how this computer language worked and how to download data from the Jet Propulsion Laboratory and all this stuff. And I had to just kind of figure it out with some help from my research mentor. But that was when I was like, this is maybe more fun than doing physics. Maybe this is what I want to do with my life.
And so did you go straight into the professional world saying, “I am a software developer”?
No, it was more like I’m not ready to go to grad school for physics yet. But I have this other skill now that’s pretty fun. Maybe I’ll do that for a while until I go back to grad school. And then I just never went back.
What was your first job?
I got hired by Bloomberg. They recruited on campus, and I ended up moving to New York and working on financial software for the first nine years of my career.
Very similar to our story, actually. I think on-campus recruiting, you know — it really works, right? I mean, you’re there, you meet these companies. I don’t know how you feel about it personally, but I feel like it’s a great foundation to work at one of those companies, you get exposed to so many different things. Okay, so tell us the story of how you go from working at Bloomberg to your first exposure to Ruby.
The first open source experience I had actually, it was Perl. I think it was Perl. And it was like a very early form of a wiki. So this is like, I don’t know, 1997, 1998 maybe. And we really had a need for team-level documentation at Bloomberg. So I just threw one together and hosted it on the server under my desk. Stop me if this sounds familiar at all, hosting internet sites on your desk computer. And then figuring out how to rsync it across our data centers, I was like, oh, well, now I have all this, you know, rsync capability. Maybe I should open source that. And so I actually sent a pull request back upstream. “Pull request” — I emailed the patch saying, you know, I built this stuff. It got merged and I was like, hey, that’s great. That was my first open source contribution.
Then years went by before I did it again. And by that time I had left two more jobs. I was working at a high-performance computing startup, and we needed to show off our ability to parallelize stuff. And so we basically just wanted some kind of a lightweight visual. And I was like, I don’t know, what are the kids using these days? I’ll try PHP, and I tried it, and I hated it. So everyone who reads your newsletter will now hate me because I hate PHP. But I really didn’t like it. Then the second thing I tried was Rails, which had just come out, right? This was like, I don’t know, 2004 or something. There wasn’t even a 1.0 yet, I don’t think. And I tried using that and I was like, oh, this makes sense to me. And I never really looked back. So that was the first time I used Ruby to do anything real.
Did you get involved in the Ruby community from the beginning?
A little bit. So the computing startup kind of fell apart, and then at the next company I went to, I had enough experience with Rails where I was like, “We are going to build our product on top of Rails.” And it was an internal-only product, but it was Rails and I was using, like, all of the JavaScript coolness to make some numbers monitor, ‘cause it was a dashboard. Of course, everybody’s got a dashboard. The numbers were lighting up and they were like the yellow tinge, if you remember the early JavaScript plugins for Rails. And we were using the Ruby gem mechanize really heavily to scrape. It wasn’t illegal scraping of data, but we needed to pull down a lot of energy data for the company I was at. And everybody had a different XML format, and I was using mechanize really heavily. And there was a feature that was missing, which was client-side support. And so, again, I emailed the patch to Aaron Patterson, and I was like, hey, dude, really love mechanize. I’m using this patch locally. Maybe it’s useful for you. And he merged it. And that was like... Aaron, you know, Aaron is very enthusiastic. And he kind of pulled me in. He was just like, hey, man, that was great. If you ever want to do anything else, please let me know. And that was when it really clicked, and I started getting more and more involved with the community. And like, look at me now. I’m the maintainer of mechanize now, 20 years later. So you know, hat tip to Aaron, where his enthusiasm really pulled me in. I’m always going to be grateful to him for that.
Being on the other side of it now and being a maintainer of so many gems that are so widely used, nokogiri for example, how do you think about how AI has changed the community work of the maintainer? Because from a human perspective, you emailed Aaron, he merged it, and now that’s someone you know, and from his perspective you’re now someone who maybe is going to make another contribution. How do you feel like AI has changed that?
Good and bad, like everything else, I think. I think the good bits are that maybe small bug fixes are more likely to get fixed by someone, right? Because they’ll be in Claude and Claude will probably tell them “hey, listen, the problem you’re having is actually this bug in nokogiri. You know, here’s the three line change. Would you like me to send a pull request?” And maybe they go like, Y return, like Homer Simpson, just hitting Y over and over again. And then the pull request shows up. And if it solves a real problem and it’s clean, there’s no real reason for me not to also hit Y and merge that.
So on the good side, there’s that. Maybe the optimist in me wants to believe that’s the case. But I think the bad bit is, engagement is really low. So where before if somebody sent a pull request and I had a question or feedback, maybe 75% of the time I’d get engagement and a response. Now, my expectation is that I’ll get a response 25% of the time. It’s flipped.
Somebody opened a bug report, I can’t remember which project it was. It was one of the tailwind CSS gems that I maintain, where somebody had sent in a fix. And I was like, I don’t really think this is fixing what you think it’s fixing. It wasn’t obviously AI, but I was like, I think that this problem you’re describing, if I understand correctly, is this and what you’re doing won’t solve it. But please tell me if I’m misunderstanding and let’s try to solve your problem. And then a month went by and there was absolutely no response from the person. So I closed it. And I think that’s becoming a lot more common. Where now I’ve spent more time on that ticket than the person who submitted it.
And that inversion, I think, is (evil foreboding music plays) maybe the lead in for more intense developer burnout, maintainer burnout. So I find myself doing two things. Number one, not being as engaged a lot of the time, because I have low expectations that the person will engage in return. Then the other thing is that I will far more often now just finish the PR and merge it. Where I used to want to be like, this is somebody that I can convince to be a repeat contributor and pull them in. So I’m going to give them helpful feedback and encouragement and see how they respond. But now I’m more likely to just be like, you know what, thanks, this is pretty good, add two commits, merge. So now I’m disengaging in response to my expectation of other people not being engaged. And that’s not greta either. But I have to kind of triage where I spend my time. I see myself doing that, and I’m not really happy about it.
Is any of that on your side because you’re also hitting Y in Claude, or are you doing those commits manually? Has the barrier for you to just finish some work that someone else started going down?
Yeah, sometimes. I think that I’m struggling a little bit to use LLM agents effectively in open source. In my day job for 37signals, I think it’s great. Because I can use it, and it pulls in logs from production. It’ll pull in exceptions from Sentry. It will even do a read-only database query to make sure that we’re talking about the right user for example. It’ll formulate a plan and it will generally execute on that because the code is structured and we have agent files and everything.
But now in my open source projects, I don’t have a lot of that supporting structure and I find that the agents flail a little bit. And so I end up doing a lot of stuff in my open source projects manually. I’m trying to turn the dial on that, but right now I usually end up pretty unhappy with LLM output on my open source projects. Some of that is I need to invest in agent files. Some of that is I need to have a style guide. All those supporting structures need to be in place, but they take time to do and I’m just spread thin.
Some people write software because they want to solve their own problems, or they just love writing code, or there are all different kind of motivations. And I could see someone saying in my professional job, I’ll use it, but when I’m working on open source, I just want to write for fun. How do you think about that?
Yeah, I mean, a little bit of both. I do like solving problems. It’s why people do the New York Times crossword puzzle, because it exercises their brain and gives them a dopamine hit, not because they want to fill in a letter in every square, right? You know what I mean? The process is the outcome a lot of the time. And I do feel that way, but also I feel like my teammates and I at work are using AI so effectively, I do see the advantages, which is you can parallelize a lot of work. You can move really quickly. So I see the upsides of it as well. And a lot of maintenance work, especially security — I could talk to you for an hour alone about security. I feel like AI is better most of the time than I am doing it by hand. And so that’s a good trade off too.
Can you elaborate on that? What do you mean by better?
Sure. Better is not the right word, thank you for poking at that. I think there are at least two ways that AI can be valuable when working on security. Number one is a lot of security is repeatable process. I’m currently in the middle of going through nine vulnerability reports that were submitted by AI to nokogiri. And they are all weird. They’re edge cases where like, if you call a protected method and pass in the wrong thing, you can get it to segfault. OK, well, why would you do that? Is this really a security problem or is it just a bug? But like, I’m not even going to argue about it, I’m just going to go fix it. But I’ve got a process, which is first, let’s write a test that we can run with ASan or Valgrind to reproduce like invalid memory read.
And then I have a bunch of prompts that I use for security work that’s say, let’s explore the solution space, give me three ideas, give me the pros and cons. Okay, let’s pick one, let’s execute on it now. I use Claude, so then I have it ask Codex to do a code review and a security assessment on it. And all of that stuff is like, I could do it myself, but it requires having to switch context from the attacker mentality (how can I break this?) to the maintainer mentality, which is how do I fix this? And I feel like I in particular fall into the hole of I’m fixing it and I’m optimistic that this is the right thing to do. And I’m not thinking critically enough and I’m not thinking negatively enough. So I like having one or two LLMs that I can just say, tell me what I’ve missed. See if you can still break this. What other attacks are in the immediate area? And they can draw from the internet and they can actually a lot of the time come up with twists or edge cases or different attacks that I really appreciate.
So on the one hand, there’s repeatable processes and mental switches that it’s really good at. But the other one is just straight up like fuzz testing. You can just point an LLM at a code base and be like, I don’t know. We’re doing a lot of stuff with HTTP headers. What are we doing weird? And it’ll be like, oh, I found this spot in Rails where you’re splitting a new line, but not on carriage return or not on line feed, whichever one that is, you know, \r \n. And you’re like, oh, great. I never would have seen that in 100 years. Great. Awesome. And all it’s doing is pattern matching. And so it can actually replace humans for a lot of that stuff. Maybe those are vulnerabilities, maybe they’re not vulnerabilities, maybe it’s not useful, but they are still better than humans at some aspect of that.
I think that’s really powerful. A lot of what I read in the headlines for supply chain attacks and open source vulnerabilities right now are much more human-oriented. It’s like somebody got a malicious version of an NPM package published because there’s no two factor auth or something. I feel like those are two different categories of security problems.
So yes, you’ve popped the bubble. The question is, what is the impact of the security fixes, right? We had some folks from Ruby Central who had access to, it might’ve been Fable, it might’ve been Mythos, I’m not sure, some Anthropic cybersecurity model. And they found a thing in nokogiri and wrote a blog post about it. But the thing they found was, again, one of these things where nobody would actually do that. It was like a regular expression denial of service bug. So number one, no Rails app is vulnerable because Rails sets the ReDoS timeout. And number two, it’s like I’m passing in a CSS query. And what nokogiri does is it takes a CSS selector and turns it into an XPath query. There’s one part of that where it uses a regular expression, because that’s how the parser works, partially based on regular expressions. And so if you feed in a degenerate CSS selector, you can denial of service attack yourself. But who’s passing an untrusted user input? And so like, great, you found it, I’ll mark it low, I’ll fix it as a security thing. But what happened there is that person spent time finding it with the model, I spent time fixing it, and at the end of the day, nobody is actually more secure. So there’s an element of security theater to all this that really annoys the hell out of me. And yeah, I’m way more worried about social vectors.
The motivation piece of this is interesting, too. As an individual, it’s exciting to potentially be the person that discovers something. There’s professional ramifications for that as well. But then there’s also the corporate side of it, where and again, AI has lowered the bar to produce this kind of content, but people are always looking for content opportunities. And so I feel like there’s this never-ending push for what can I write a blog post about? Well, I discovered a potential vulnerability. How do you think about that kind of thing?
I’m going to put on my glasses that say, what is your incentive? What is everybody incentivized to do in the system? Of course if you’ve got a bounty program, people were already incentivized to throw crap against the wall and see if it sticks. And now with AI models, it’s even cheaper and easier for them to more efficiently generate more crap. And I’m sure you’ve read all of the same blog posts I have, from the curl team and everyone else. But an interesting thing that happened in response to this that I don’t think got very much publicity was the internet bug bounty shut down. HackerOne used to run this thing called the Internet Bug Bounty where they got checks from big companies every year. And they would portion that out to reporters to open source projects. Rails participated in this, which meant that for every report that was published and fixed, the reporter would get 75% of a bounty and 25% would go to the maintainers, which really meant it just got funneled into the Rails Foundation. And nobody who actually maintains Rails actually saw a dime of that money, I don’t think. But the idea was that was supposed to incentivize good behavior from people. And HackerOne was just like, this no longer makes sense because it’s so cheap to generate new reports. We’re going to overwhelm the system. And so instead of trying to throttle it down, they just canceled the whole thing. Maybe we’ll come up with a new system later, but for now, it’s done.
But maybe the really interesting thing, as far as I can tell, is that the reports did not slow down when the internet bug bounty shut down. If anything the Rails team is getting more now than we were earlier this year when the bounty program was still in place. Like, it’s really nuts. It’s insane, and it’s overwhelming.
Do you think there’s a temporary period here where we’re going to shake out a whole bunch of bugs in libraries over the next year, and then it’s like, first we invented fuzzing, we fixed some stuff, and then we invented this, and we fixed some stuff, and then all these established projects will be secure to the kinds of things that Mythos can find? Or do you think that forever the world of open source security is changing?
Yeah, I don’t know. That’s a good question. If I had the answer to that, I would maybe do something differently. I will say that the reports have changed a little bit where they are not so easily detectable as BS anymore. What I mean by that is they’re better written, first of all. It used to be they would come in and it would be garbage and it would even still have part of the prompt that the person gave to ChatGPT, or ChatGPT would say, “If you give this, you’re 20% more likely to get a bounty.” And they’d just copy paste that and stick it down there too. Really lo-fi stuff. So they’re getting much better written now.
Some of the recent reports we’ve gotten in Rails have been about the development environment. I’m in development, and Rails turns on some features like exception reporting. So now you’ve got a stack walkback in your screen. And then some of those exceptions can even have an action that can get taken. And there’s a link that you can click on the link and it will go run the migrations for you, for example. So then people will be like, okay, well, that is only enabled in development on like localhost or 127.0.0.1, but I can trick it into doing it on a different port, and they submit that. What do you do with that as a team? It’s not obvious baloney that you can just close and say, N/A. And it’s not obviously something that’s like, oh my God, we need to fix this and we need to have an embargoed security release for Rails. It’s somewhere in the middle. And that’s really what I’m seeing, more and more stuff that’s like, maybe it’s ticky-tack, it’s not high impact, but it is something that needs to be fixed. It’s in this middle range. Maybe it’s a normal distribution. But it’s getting harder to tell something that’s in that middle from something that’s really severe quickly. And so again, it’s very easy to get overwhelmed.
And they’re presented as more severe because of the incentives.
Always. So the question you asked was is this a trend or has security changed forever in open source? I don’t think it’s been a phase change of matter necessarily. But I think that if I was going to bring somebody on as a maintainer onto any of my projects tomorrow, I would probably look more at their security know-how than their taste or their ability to write good tests or their communication skills. That’s becoming a skill that is much more in need than the rest of it. So maybe that is a phase change, where it’s going from being a community thing to being this defensive posture where you’ve got to figure out how to put up the wall and filter effectively and be responsive.
I wonder if there are maintainers out there who’ve maybe received one security report in their entire history of maintaining because no one was pentesting their library unless something was obviously really broken. But now you might get real reports that are medium-severity vulnerabilities. And you’re just not used to handling that flow. People need to be trained on that.
Yeah. I had HackerOne programs for nokogiri and loofah for years, and I got single-digit reports over a time span of years. Meanwhile, in the last two weeks, I’ve gotten 17 reports across nokogiri, loofah, and mechanize. That’s weeks of my time to go through that now. Thankfully, I work for a company that’s like, yes, we hired you to do open source some of the time. So go take care of it. But it used to be I would get one a year. And now I’m getting a couple every week, at least.
Is there anything we can do about the burnout question? Have you found anything that helps?
Yeah, thank you for asking. So first of all, I’m one of these people where I try to be responsible, which is how I always get into management roles in most of my jobs, right? It’s because I’m just being the adult in the room and trying to do the responsible thing and they’re like great, you’re in charge. I got into security the same way. I feel a strong sense of obligation.
So when I started working specifically on the Rails Security team last year, relatively recently, I got to see up close how real security experts deal with this stuff. And the number one thing I think that maybe I still need to internalize is 99% of the time, things are not on fire. It is best to take a beat, take a breath. Step back. Ask about attack vectors. Ask is this really a problem? If we’re going to fix this, how would we fix it? Can we fix it without a breaking change? Does this need to be embargoed and a security release dumped on people? And so I watched the Rails security team. This is people like John Hawthorn and Aaron Patterson, Matthew Draper and other people. They sit back and they’re fine sitting with it for a bit of time to figure out what’s the right thing to do. And my reaction is always like, we’ve got to fix this as fast as possible. And I think that makes it harder for me and more likely that I will burn out. So that’s number one, not every security report is a friggin’ emergency.
But that also becomes hard because a lot of the reporters, especially if there’s a bounty program, this is how they make their living. So they’re going to pester you for a response. There are people who go on the HackerOne report or on the GitHub Security Advisory once a week, and they’re like, “Hey, just pinging, what’s going on?” “Hey, this has been a month. I haven’t heard back from you.” “Hey, if you don’t get back to me at the end of this week, I’m going to go public with this.” And that stuff I think, is — I understand the incentives that lead to that behavior, but it’s really unhelpful to have that pressure on the other side.
So maybe I didn’t answer the question directly, but the thing that I’m trying to do to make it more sustainable for myself is to slow down and understand that it’s not an emergency and try to do it effectively. And that sometimes means saying hey, I’m getting closer to burnout. Maybe I just need to step away from the computer for a week or just not look at security stuff for a week or something. And I think people feel that way. There’s this guy, Hal Brodigan, who is known as postmodern, is his handle. And he’s been leading the bundler-audit project and the Ruby Advisory Database project for years. Thankless job. And recently he just stopped responding. And none of us really know if he’s okay or not. I hope he is okay. And if you mention this, just say that, like, Hal, I hope you’re okay.
Burnout’s fascinating too in the sense that at least personally, the times that I’ve been burned out, it’s not necessarily always from too much activity or stimulation, that’s often it, but sometimes it’s just not the right form of stimulation. Or sometimes you can kind of get your spark back just by finding something you’re really curious about, or is intellectually challenging, something like that. And then you will pour a hundred hours into that thing. Burnout is kind of interesting in that way in that people have a very big capacity to do work if it’s something that they are genuinely interested in and challenged by.
I keep a copy of Christina Maslach’s original burnout paper on my hard drive, and I re-read it when I start thinking about burnout and stuff. If you haven’t read it recently, it’s worth a read. She talks about several elements of burnout. A mismatch in expectations. A mismatch in workload. You feel like you’re working more than you want to. Check, check. A mismatch in control. Again, check check — you’re in reactive mode. A mismatch in community engagement, which we talked about. And fairness, like it just doesn’t feel fair, check. It doesn’t feel fair to have people create work for you. And mismatching values is the sixth one. I think if you look at all of those things in relation to open source maintainers you think oh, yeah, that’s pretty bad. I guess my real worry is that I don’t see any of them trending better, systemically. And that’s why I’ve gotta remind myself to calm down, take a breath, not a big deal. Don’t need to respond to it immediately. I’m trying to be a little more long-term oriented. That’s all.
Yeah, fascinating. That’s something at RubyConf we plan to talk with others about as well. Where do we as a community go from here knowing that this is a risk and especially in the age of AI, how do we keep up? Thanks, Mike.
One last note. I went back and I looked at the e-mail I sent to Aaron for my first mechanize patch. And it was 20 years ago next week. June of 2006.
Once a Maintainer will be going live at RubyConf in Las Vegas on July 15th! Join us for our conversation with Jeremy Evans, Richard Schneeman, and Colby Swandale.
For questions or to suggest a maintainer to interview, write to Allison at allison@infield.ai.