This is the GitHub Podcast, a show

dedicated to the [music] topics, trends,

and culture in and around the open

source developer community on GitHub. My

name is Abby. I'm on the open source

programs team here at GitHub, and I'll

be your host for today. In December

2021, a critical vulnerability was

discovered in the open source Java

logging library, Log. The incident that

became known as Log for Shell allowed

attackers to remotely control affected

systems with just a single line of code.

Its impact affected millions of

applications worldwide. Luckforelle was

actually one of the first libraries I

ever used when I was learning to code.

So seeing it at the center of this

massive security crisis really hit home

for me. In today's episode of the GitHub

podcast, we're exploring the aftermath

of log for shell and how it

fundamentally reshaped how governments

and organizations approach open source

funding. And if you want to know more,

be sure to check out the video the team

here at GitHub recently published about

the maintainers perspective on that

crisis. We'll link it below in the show

notes. So just to set the stage, we want

to talk about what happened after log

for shell and how this really shaped

what open source funding looked like in

the future. And today I'm joined by two

wonderful guests. We have Felix Rea, the

director of developer policy at GitHub.

Hey Felix.

>> Good to be here. Hi.

>> And we have Christian Grommyer, a log 4J

maintainer.

>> Hello Abby.

>> So can you do brief introductions of

yourself? uh maybe start with Felix on

just your experience at that time.

>> Sure. Yeah. So before I joined GitHub, I

was active in various open source

funding efforts over the years. And uh

at the time of lock for shell, I was

lobbying the German government to try to

invest in open source maintenance uh in

my function as a board member of the

open knowledge foundation Germany, which

is a German NGO that supports open data,

open access, and the open movement in

general.

>> Nice. And Christian?

>> Yeah. Um well I'm a self-employed uh

yeah well trainer and software developer

and consultant these kind of things. Um

well and I'm also involved in the Apache

lock forj uh team and when lock for

shell hit I was actually sitting on my

table and was wondering what was going

on with my email inbox. So

>> do you see the numbers just ticking up?

>> Yes. And it was not very comfortable to

see. So, so I'm very bad with my emails

anyway, but this time was different.

>> Oh, I feel that the stress of the email

numbers. Can you take us back to that

time? Like what was the mood in the

government policy circles and did this

event change anything? Yeah, I think log

for shell for the first time showed to

people how important open source

maintenance really is. Like um open

source software has been compared to

roads and bridges, you know, critical

infrastructure and kind of in the same

way that we don't necessarily spend

enough on our physical infrastructure

until a bridge collapses and there's a

lot of damage. It was kind of similar

with lock for shell in a way because uh

the effects were immediately visible to

normal people. So just to give you an

example in Germany the German cyber

security agency put out a red alert and

uh what a lot of I would say sensible

organizations did was to shut certain

services down just to prevent any data

from leaking. For example, the German

health records were just taken offline

for a short while. And so even

politicians who have no connection with

digital policy notice that there's

something going on and this is something

worth paying attention to.

>> And I know you were involved with uh

founding the sovereign tech fund. Can

you tell us a bit about that and how

yeah how this came about

>> around the log for shell vulnerability

becoming public? This was just after we

had elections in Germany and uh the

proposal for the sovereign tech fund was

actually already there. So we had worked

with uh the government before that to do

a feasibility study and kind of explain

why this would be a good investment and

we had uh a really great champion in the

German government uh in the agency for

disruptive innovation called Sprint. So

this is a government agency that kind of

works a little bit like a startup

incubator but for public sector projects

and the head of that agency called

Rafael Laguna de la Vera is kind of an

old school open-source um advocate. He

was all involved in Zoua back in the day

and uh so he saw this initiative from us

and said hey I want to help you make

this a reality and so he put us in touch

with people in the government to do this

feasibility study. But then of course

there were elections, new government and

we thought okay great we have to

convince them all over again that this

is something that's important and I

actually remember I pitched this to a

journalist like we had the Finnish study

and I talked to a journalist at Deshbiga

and said hey don't you want to write

about this and he said well thanks a lot

but this sounds a little bit boring like

something like that. Um, so he kind of

kindly uh turned me down and then a few

days later lock for shell happened and

he wrote this amazing article. Um, I

think it was called something like how

do you put out a burning internet

because uh the crowd strike founder at

the time had some said something about

the whole internet is on fire or

something like that. And that article I

think did a lot of the convincing

because he was saying like hey there are

actually um proposals on the table on

how to better support open source

infrastructure. this is not the first

time something like this has happened

like that he was pointing to heart bleed

and I think that helped um to convince

the new government to actually pick up

this initiative and then on the day

before Christmas the uh ministry of

economy of the new government announced

okay we're actually going to fund this

proposal and create the sovereign tech

fund

>> it's amazing merry Christmas to everyone

>> well [laughter] it was probably a less

uh exciting Christmas for you.

>> Well, it was different. I I think I was

one of the last persons to hear about

the STF fund. Uh and when

well, when I heard about it, it was like

uh a friend called me and told me, "So,

did you hear about the STF?" And it was

like, "So, no, no idea what is that."

And he said, "They they giving money to

open source projects, money like your

project." Actually there was this rumor

uh the STF was funded uh because of us

and they said wait wait wait how is this

possible? So there is an organization

and it came into existence because of an

issue we built and nobody told us and

this was like so yes probably it's like

that and uh did you apply and they said

I never heard of this how could I but

anyway uh long story short I applied and

it was like uh the director was like so

hey didn't we ask you to apply before no

it seems somebody forgot So, so I don't

know how this really was. Anyway, uh we

applied for the STF and uh yeah, they

granted granted uh us the money.

>> Yeah, this is amazing to me to hear

because I mean even though the proposal

was out there, log for Shell definitely

was sort of the catalyst that helped

convince politicians, hey, we should

actually do this. It's yeah, amazing to

me that uh nobody reached out to you in

the beginning, but you know how it is

with these new projects.

>> Yeah, it's definitely a pattern I see

where open source maintainers don't

realize what funding is available to

them or what options are available to

them. Christian, I know you mentioned in

the video that um in reality only two of

the 10 maintainers for log forj was

actually able to accept that funding.

Can you walk us through why?

>> There are many reasons why uh you would

not accept money. So imagine the

following situation. So I I had this

application uh it was accepted. I got a

bunch of money really a lot of money and

then I went to my team and told them hey

everyone I have money who wants it. And

then the reaction was oh well I just got

this new job. I can't take it because

you know it's safety. And the other one

said something like uh well taxes are

complicated. Taxes are really

complicated. I I don't want to mess with

that. Um couple of people were just like

no I I'm not interested to work in open

source. So uh the team members I could

ask went down and went down and finally

uh two people joined me in this project.

uh one uh Potra uh comes from Poland and

he was looking for a change anyway and

he was very glad uh to see this

opportunity and the other one Falcon he

uh is really very very good uh

developer. He was also you know very

excited for the opportunity to really

work on lock forch and make it better.

So they were convinced and without the

funding they would never have taken this

opportunity but all the others they had

so many reasons not to take it. Yeah, I

think that really highlights how funding

is only a small part of what we need for

open source sustainability and to secure

the open source ecosystem,

>> right? And actually when I when the

funding was running um and the money

came in, there were so many questions

for me as well to you know how to

distribute this money, how to pay all

the taxes and then suddenly we have so

many organizations in Germany. They see

you're getting a lot of money and the

first thing what happens is they're

knocking on your door and saying hey see

you earn a lot more so give me your give

me a share of it and then it's for the

developers it's not for you but they

didn't listen to me so I had to share it

with all the other organizations as well

>> yeah it's really interesting to hear how

money can sort of change the dynamics of

an organization I've also seen this

sometimes like in some NOS's that I've

been involved in where in the beginning

everybody is a volunteer and then after

a while you become more

professionalized. Maybe you get some

funding and then some people start being

full-time advocates in the organizations

and others continue to volunteer and

sometimes it changes like that the

volunteers end up doing less than they

did before because there's now somebody

whose job it is to do these things and

so it can I think it's important to

think about that and to also help

projects uh come up with yeah ways to to

use funding that are beneficial for

everybody or at least to be aware of

those changes and

>> uh how it can affect people in the

project.

>> Yeah. Yeah. I've definitely seen some

projects it just implode when they have

this money or they just sit on that

money for years. They don't know what to

do with it. Uh if it's anything more

than like pizza party levels of money

it's or stickers level of money. Um it

becomes a big issue in projects. So I'm

glad you were able to figure out you

could fund people Christian and figure

out what to do with that money.

>> Yeah, that's thanks to to my colleagues

as well. So because they were all very

pragmatic. So you must imagine we we are

coming from different regions. So

Netherlands, Poland. So they're all

having you know um different expenses

and and cost of living is is very

different. then and then you have to

find a solution to pay a fair amount to

everyone so they can live uh you know

let's say good

>> um but also it would mean that example

the the rates in Poland would be lower

than the rates in the Netherlands so but

we figured it out because we were all

friends and uh but yeah well in the

beginning especially when we didn't know

each other so well it was a little bit,

let's say, more stressful.

>> So, I'm hearing a couple things we've

learned from this whole situation.

First, uh, is communicating with

developers that there are opportunities

for money for them. Um, then also just

having the correct guard rails in place

when you do have money uh, so that your

project can grow and flourish. What else

have we learned through this experiment?

What I have learned is that uh when you

give one group of people money and the

other group of people who declined the

money do not get this money. So then you

have two groups and they are somehow

different and this builds up some kind

of a tension I would say and uh because

previously everyone was like we're all

equal and now it's not longer really

equal because some people have the

opportunity to work let's say full-time

and others uh they cannot do this. So

what they can do is they can just you

know watch the project grow and change

but they don't have the time to follow

all these changes so they feel excluded

in a way

>> and what I when I saw this uh tension

rising I tried to communicate not only

with the people who you know earn this

money and and work for this money but

also with those who are not affected by

the money at all because we need to

include them and uh that was really

complicated I have to say uh because

there are so many emotions about it. So

imagine the person who funded founded

the project uh he could not take the

money. So it's literally his baby.

>> Uh and now we're taking over with a lot

of money and a lot of energy and a lot

of you know let's say time. Um I don't

think this feels really good but again

we all sorted it out and with talking to

each other we could you know um make it

uh really you know very well and very

comfortable for everyone. So eventually

uh we grew was the project we grew as a

team

um but there were definitely the

situations so we could have gone you

know the wrong direction so to say

>> and then Felix from your perspective I

think you've seen a more broader view of

this funding across projects um what are

your takeaways after seeing this? Yeah.

So with every funding initiative I've

been involved in there has been some

sort of lesson learned. So maybe even

before the sovereign tech fund I was

involved in uh starting an EU pilot

project for open source security called

uh FASA and in that case the big lesson

was trying to work around some elements

of government bureaucracy. So there at

the time we had the problem. Okay, we

had approval for the European Union to

spend some money on open source security

but because of public tender rules they

could not give the money directly to

developers and so we ended up doing a

buck bounty project because we thought

okay in this case we can give the money

to a buck bounty platform and then the

buck bounty platform has won the public

tender and they can then distribute the

money to developers. But of course we

learned bug bounties are not the best

way of supporting projects and

especially if you don't also uh

incentivize the fixing of the bug. You

might just create more work for the

maintainers. And um I think we all know

this nowadays that you can't just you

know start a bug bounty and expect great

things to happen like it can even be

counterproductive. But that was sort of

one lesson and um now kind of hearing

this and also um kind of looking at my

time at GitHub um the XZ utils uh was

discovered the first week I think I was

at GitHub or maybe the second week. So

it was really new and I think um it has

really shown this uh aspect of community

health and how important that is and I

think what Christian has explained is

pointing in a similar direction that a

lot of um open source development is

kind of yeah a human sport in a way and

uh we need to I think pay a lot more

attention to supporting community health

as well and that can be through things

like giving people opportunities to meet

in person. I mean, especially with these

globally distributed teams, I think

that's something that makes open source

software really great, that you have

people from different countries working

together on something that uh they're

passionate about. But yeah, it can

sometimes be difficult if all your

communication is kind of through email

and chat and things like that. It's

easier to have misunderstandings. And so

I think um

it's useful for different funding

programs to also build community. And I

think that's like something that

>> uh I've really enjoyed seeing in the

GitHub secure open source fund that the

uh developers who participated in that

continue talking to each other and

exchanging ideas and um yeah hopefully

that can also help grow those projects

and like bringing in new people who are

interested in supporting them. what you

were talking about reminded me of this

book I read years ago uh Twitter and

tear gas by um Zanfei she's a researcher

and journalist but she was saying just

in this age of this digital age it's

really it's really easy to spin up quick

movements or like bring people together

quickly to do something but then they

haven't done that hard work of working

in community and figuring out how they

communicate and how they solve conflict

um so it becomes very fragile and it can

fall apart pretty quickly um and I I

think it's I see this in these open

source communities like we need people

who like work together really well and

build that strong community so they can

withstand getting tons of money and

figuring out how to move on but if they

are just coming together yeah really

quickly it's hard to build that muscle

and it's hard to understand how to

disagree with each other and then still

move the project forward. So I am really

glad that log 4j was able to do

something like that Christian but it's

not it's not true for every project that

I've seen. Yeah and and you need to know

that lo forj is a very very old uh

project. So we we are 25 years old

>> and uh even lo for j 2 where majority of

the team members uh changed or you know

went away. So it's still years and years

old so we know each other very very well

now and we respect each other and so

what you what you're basically saying is

when people come together

uh very quickly they don't know each

other. So there is no real respect. So

respect is something that needs to grow

I think and when you work together for

like 10 years then of course you you're

respecting each other in a way and when

something extraordinary happens then you

can you know fix it together.

>> So we talked a lot about lessons learned

and things to focus on. Felix, you

actually mentioned the GitHub secure

open source fund, but Christian, how how

was your experience with that? And how

would you Yeah. Would you call that a

success story for Log 4J?

>> Uh, yes. I I call it not only a success

story for Log, but also one for me to be

honest. So when I got the invite to join

this uh program, it was like maybe the

mixed up uh emails. So this is not for

me [clears throat] because I have no

idea about security something like this.

And uh then I joined uh this program and

I was seeing so many other developers

having the same problems like me and

they're asking the same questions. And

another thing I found out was like um we

are not doing so many things wrong. So

we really so the the best practices

explained there we actually did a couple

of them very well already but it always

felt like we doing too little. Now we

have the confirmation that couple of

things went very well and the rest we we

now know where to uh put our hands on

and how to fix it. And so for me

personally, I felt more confident

because I learned something

>> and on the other hand, I I didn't feel

like uh you know, I had no clue anymore.

I now could you know uh think about

myself. So at the level that I am and

well for lock for we actually

implemented a couple of things. So the

the missing pieces so to say and have

some initiatives uh going on to improve

the rest as well. It's funny that you

mentioned that that um you didn't think

of yourself as a security person because

like for me I'm not a developer at all

and I um just took the opportunity to

listen in on a few of the lessons or um

uh sessions in the uh secure open source

fund and I found like oh I can learn

something here. uh for example there was

a session on how to write a good CVE and

I found the skills needed there have

nothing to do with software development

it's about communicating clearly and

like hey this is something that I as

like a communication science major can

actually resonate with and so that was a

good experience because like sometimes

as a non-developer in a very developer

focused organization I don't know

everything that's going on and I need to

uh brush up my skills as well so I I

really enjoyed being able to listen in

on that.

>> Yeah, I listened in on a few sessions,

too. And it's been uh No, it's been

really valuable. So, I'm looking forward

to seeing more of that uh released

openly.

>> Yeah, one thing that I was sort of

worried about like um

when we designed the sauce fund was this

idea of creating a curriculum because

especially sort of having this

experience with the buck bounties,

right? I was like, "Oh no, are we

creating even more work for

maintainers?" So, um, yeah, I was kind

of curious how this was going to land.

Like, first of all, is the program going

to be valuable to people? Is the funding

a good incentive to participate and so

on? And um, yeah, I hope we landed that

in a good place.

>> Yeah. And Felix, I know both of us were

involved with the creation of the SAS

fun, just influencing how um, how it was

shaped. And I know for me, the part that

really resonated was that cohort model.

And Christian, I'm so glad the community

really helped you because uh I think

anytime we can build more community

within open source, it's better for the

ecosystem.

>> Yeah, absolutely. And what was uh very

special in this cohort was uh we were

one of the few, if not the only Java

project. And that was very interesting

for me as well because as a Java

developer, you're usually looking at all

other Java projects. But there are so

many Python projects or even uh you know

uh enduser software projects and it was

like oh they have problems I never

thought about. So like people who who

are creating UIs for end users they they

need to think totally different than

library developers like we are. And so

this alone this uh diversity you created

this was so helpful and interesting and

open my view um you know of uh from of

community.

>> So Felix beyond the secure open source

fund how has all all of these lessons

how have all these lessons you've

learned um shaped your approach to

developer policy here at GitHub?

>> Yeah. So one thing that I've sort of

found over the years is that we really

need a diversity of funds. So it's not

like

public fund versus private fund or

something like this. I mean GitHub

secure open source fund is an industry

fund. Different companies pay into it

and then GitHub runs it. Uh sovereign

tech agency is fully funded by the

German government. Um but there are

other examples as well like for example

the prototype fund in Germany which is

also funded by a different part of the

government funds completely different

things like um where uh the sovereign

tech agency focuses on the maintenance

of really critical infrastructure. The

prototype fund is really about

incubating really new small projects uh

teams that are just starting out and

want to spend six months just developing

an idea and um I think it's really

important to have this diversity also

the open technology fund in the US NLnet

all these different things because no

fund is going to be right for every

project um the needs are just really

different like for example there are

going to be uh projects that, you know,

need a lot of people to work on it

full-time, whereas you're going to have

others that um maybe need support with

uh winding down a project or recruiting

a second core maintainer or things like

that that are not necessarily even

funding related. And um so I'm currently

at GitHub advocating for the creation of

an EU fund. Um we have uh financed a

study done by open forum Europe um for

basically the feasibility of an EU

version of the sovereign tech fund and

we're taking all of these lessons and

one lesson I think that is important for

me is um I have really learned a lot

about the role of industry in all of

this. So in the beginning I mostly was

um thinking of you know these great

examples that the curl developer Daniel

Stainberg is always doing in his

presentations like some big company

asking um users to you know ask their

tech support questions to the curl

project which is then essentially one

person. Um that is obviously bad

behavior but what I didn't know before

is how much companies do uh invest in

open source maintenance in the form of

developer time or in the form of uh

funding open source foundations and all

these different ways but it isn't

necessarily distributed in the way that

it reaches the projects that need it

most. And so I think um if we can do

more research into what works and what

doesn't work in all of these different

uh funding experiences

um we can yeah build sort of the next

generation of uh open source funds that

are hopefully going to be even more

helpful.

>> And then Christian from a maintainer

perspective what approaches are you

seeing now that give you hope for the

future maintainers or the next log for

shell? Well, actually what gave me hope

is that something happened after lock

for shell and it was not just fixes. So

we have all these funds now and we have

uh politicians who look at open source

differently now. Uh we have all these

people uh trying to make things work. So

like Felix an example. So he's you know

even he's even paid for for doing

something like this. So that's exciting.

Um so but uh we will we will you know it

will take time to really uh you know fix

this problem of open source funding. So

there are so many questions. So like

Felix already mentioned so who is

actually receiving the money, who is

getting paid and who is not getting

paid, what is considered critical, what

is not and um even when you say I give

you a bunch of money dear developers,

then all developers know this money

stream will end. So what happens next?

What's with my family then when they

suddenly have no income anymore? So

there are so many uh questions and I

think we will probably need five or 10

more years uh to get this really you

know fleshed out so to say. Um but there

is movement and when I started with open

source there was absolutely no movement

at all. So open source was like uh when

I told somebody hey I'm I'm going to

work for free at night [laughter]

everybody was like are you stupid? Then

it changed to oh you are one of the cool

guys hacking at night so that's that's

nice but of course nobody would give me

money for that and today now we know in

these highly technology

you know related world that we live in

uh that without open source nothing

works here anymore. So, and this this

view has changed and that gives me hope

because now open source not is not just

for hackers at night. So, it's really

driving the world so to say. It's in

every well car probably. It's in every

phone. It's everywhere. And we cannot

even send money to our grandma without

open source.

>> Well, thank you both. Um, we like to

close off each podcast where everyone

shares an open source project or repo

that's excited them recently. Felix,

would you like to go first?

>> Yeah. So, since we've talked so much

about uh the critical infrastructure end

of the spectrum now, I'm going to shout

out more of a smaller project actually

funded by the prototype fund. Uh, it's a

period tracking app called Drip. Um,

>> oh yeah, I love Drip.

>> Oh, you know it. Awesome.

>> Yeah, I've worked with Marie before.

Anyway continue.

>> Oh, exciting. Yeah, I two things I like

about it. One is like open source is not

just a means to an end, right? Like so

there in this case of this period

tracking app, there's a really good

reason why it's open source because like

uh this is very sensitive information

like medical information that you don't

necessarily want to share with uh third

parties. So by making it open source and

storing all the information on your

device, it's very privacy preserving.

And the other thing that I like about it

is that it's not pink and it's not sort

of making assumptions about who is using

this app like it's not just women who

want to conceive or something like that.

And so it has I think put a lot of

thought into the design as well which I

really like about it.

>> Ah yeah no that was a great project to

shout out. Thanks Felix. Christian do

you have a project? Yes, I have one. But

mine is probably a little bit more

boring than Felix. [laughter] But uh

recently um because there is so many

things going on I I wanted to patch up

my website and then I was thinking well

do I will I do it like always with

checkle and a lot of handwriting and so

and then I found this ghost open-source

uh CMS and they previously I thought it

was a it was a company selling services

so to say and then I looked and it's

actually open source and you can

download it and I was a little you know

hesitated hating at first because I had

this experience with uh with WordPress

which it's it's fine but it's not my cup

of tea let's say like this and uh then I

was looking at Ghost and they had these

wonderful docs and they told me step by

step what I had to do and even when I

have no idea about how this software

works I had it up and running in like no

time. So, uh, Ghost was very cool and

they support Docker, which is also an

open source, uh, project I like. Um, and

what I specifically like on Ghost is

they are also very open when it comes to

all their uh, numbers, their financial

numbers. Even they're speaking about how

they would, you know, support the wider

community, the open source community. uh

it's not so closed closed company like

the usually closed open source companies

and um I don't know if this is true I'm

still researching but uh at least the

source code they wrote I liked it very

much and so that's my project for the

day

>> nice um for my project I went the

infrastructure route so I'm highlighting

ttcmapc

so I'm based in Toronto Canada and TTC

is our public transit system And a lot

of the information on like delays and

reroutes and disruption is found in like

different spots on the government like

the Toron City of Toronto website. Um,

but someone did this open source project

used all this open data that the city

already releases and just put it into

this nice visualization so I can quickly

see like which stops have delays, which

ones are rerouted. Um, and it's actually

very handy and I think it was created by

a PhD student at uft. So shout out to

ttcmap.ca.

CA. All right. And to everyone who's

been listening, this has been the GitHub

podcast where we talk about topics,

trends, culture, and everything about

the open source community on GitHub. So,

I've been Abby. You can find me online

at Abbyss Abbs

on most platforms.

>> Felix Rita. You can find me under at

Zenficon Senfo

N. Mostly on Masteron.

>> My name is Christian Guraya. So this is

very difficult to pronunciate but if you

search for lo for J or something similar

you'll find me and you'll find me at the

websites of the Apache software

foundation or on LinkedIn with my real

name and uh on GitHub of course with my

last name and I'm also on Master Done.

>> Well, thank you all for the great

conversation.

>> Thank you.

>> Bye everyone.