While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
Very impressed by how quickly action has been taken by this and other instances to patch the issue.
Very, seems like great work.
Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.
I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD
So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear
What I did was add my account (again)
There was no warning or anything, and it populated the list with two of me.
At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?
I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.
I logged back in, and now everything is back to normal
deleted by creator
uh, why did you have negative one dislike?
Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.
huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances)
Nope, just like Reddit it’s a value that ranges between negatives and positives. If I get two thousand upvotes, positive 2k. If I get two thousand downvotes, negative 1999 (because iirc you start with one by default).
Not exactly sure I understood what you meant by “either positive or zero”.
see your comment rn, it has 1 upvote (from yourself by default) and 0 dislike (so it’s not shown)
but in the screenshot I sent above you got 287 upvote and minus -1 downvote (making your total 288) which is mathematically correct but seems like an unintended behavior
for example this comment of mine normally have 9 upvote and 2 downvote (which is shown as a positive integer 2, not negative), making my total upvote 7
Just occurred to me that the app I use also shows separate counters. I fooled myself into thinking it was a single counter.
That’s interesting. Remember it’s a very new platform, minor bugs aren’t out of the ordinary.
I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.
I bet that little cunt Steve Huffman paid them to sabotage us
As I pointed out in the thread it was probably a few Lemmy users themselves that did it.
it doesn’t matter who, it’s the “why”. They get nothing from this, the only one who benefits from Lemmy going down is spez
I’ve sat at keyboards beside people studiously working their own. My presumption was that we were working on the same project. Then they have their AHA moment, and show me how they’ve hacked into our host machine.
They didn’t do it for money or to cause disruption. They did it to see if they could, and succeeding was reward enough. Then, happy as could be, they set off in pursuit of their next accomplishment.
Ya never know what’s going through others’ minds or what motivates them.
And some people do it for teh lulz.
Not everything has to be some conspiracy theory.
Yeah it seems like the attack wasn’t “malicious” per se in that the attacker didn’t seem to want to completely take down the site. They just seemed to want to mess with everyone by redirecting to lemon party and other mildly shocking things like racist remarks. Reads more like a bored person to me than anything.
2008 4chan humour
Gotta bring back the classics
Pool’s closed, sorry!
Nah. There’s far too much risk for Reddit to be involved. If even one hacker spilled the beans it’d cause a massive panic for Reddit investors.
It’s well established that Rdeadit* doesn’t make foolish moves. /s
I’m not saying they’re behind this. I think they are not. But I’m not ruling them out because of their acumen.
During the event a message flashed on my screen, “This website has been seized by Rdeadit for copyright violation”. That suggested two things to me. Rdeadit didn’t write that. Whoever did write it doesn’t understand how a domain would legally be seized.
“It would be extremely stupid and would inevitably backfire if reddit was responsible. Seriously, if spez has one ounce of foresight he would not be involved.”
Well you have convinced me reddit is behind it.
I thought we weren’t going to be Reddit-brained over here
Reddit, Threads, Twitter, Elon, and, Zuck seem to be all people know how to talk about in the more popular communities.
And don’t forget circle jerking about how the average IQ of a Lemmy user is higher than average.
Deleting hospital fees/debt is very dangerous… In many HUGE regions in the US there’s only one hospital and if that hospital suddenly can’t pay its bills it could shut down, leaving a whole lot of completely innocent people in a very sad, people-are-dying sort of state.
In fact, something like this already happened:
https://www.cbsnews.com/chicago/news/st-maragrets-health-central-illinois-hospital-closing/
Hospitals are special in that they’re often evil organizations (not all though) that are some of the easiest to hack but also provide critical services to the most vulnerable. One should tread lightly. Political solutions are better (hack some politicians that are against healthcare reform instead).
Clearing credit card debt via hacking is nearly impossible but I agree it would be a much more ethical choice for hackers to target. I used to work for the credit card industry. My unique insider perspective, deep industry knowledge, and personal experience is here to let you know they suck. They are just as evil and unethical and unnecessary as everyone thinks they are! Seriously: If Visa, MasterCard, American Express, and all the lesser players suddenly disappeared the world would be a better place.
Before that can happen though people need a backup payment method that doesn’t go through their systems and no: Cash won’t work (there’s not enough in circulation and it’s dangerous to carry large amounts of it). The credit card companies know this threat exists which is why they lobbied Florida (and probably other states) to outlaw alternative, government-run forms of payment (e.g. central bank currency).
As soon as people have a widely accepted payment option that doesn’t go through Visa and MasterCard’s middlemen (e.g. First Data) then hackers can take their gloves off! Until then though… Let’s keep the payment infrastructure working, OK? Thanks!
There’s no limit to the amount of good deeds hackers can do though. So let’s encourage that! For example, there’s plenty of cartels and evil religious organizations (e.g. Taliban, ISIS, Mormon Church, Prosperity Gospel scam artists) that have plenty of money to spare and enormous attack surfaces 👍
Hospitals are special in that they’re often evil organizations
Just want to state the obvious and say, this is pretty much only the case in the US.
I think the alternative payment systems in the developing countries are actually good. UPI in India is very utilitarian. China also has the wechat thing. I guess the issue with these are that they are not universal and limited to a single country.
Ribbit
clearing credit card debt, deleting hospital fees, or something else that actually serves the public good,
Inflation does very clearly not serve the public good. That aside, causing havoc in banks and medical institutions would have other unpleasant effects.
How about cleaning the bottom 10%'s debt, with the earings from one week of the top 0.1%?
Ah, you mean unauthorized “redistribution”, not unauthorized “vanishing debt”.
Technically should do less harm in terms of inflation, but money lying around is different from money being used, so there’ll still be an increase in inflation.
The part about causing havoc - kinda same, there may not be direct inconsistencies as in the initial variant, but there’ll still be some confusion due to the “top 0.1%” possibly being petty and trying to get their money back.
I frankly prefer changing the rules so that there’d be fewer artificial barriers for competition and economic efficiency to this. Say, patent law and trademark laws and IP laws have basically outgrown their usefulness and are now just a plague. Same with various licenses and practices for medical/pharmaceutical stuff (I know that things should be tested and an average person can’t tell a hoax from a normal thing, just entities doing certification shouldn’t be able to block stuff which would then be used to create oligopolies). Same with telecom. And so on.
Except for air traffic, water traffic, road traffic and radio, of course. Not regulating those would mean, eh, real havoc.
you mean unauthorized “redistribution”
Fine, let’s do taxes: how about cleaning the bottom 90%'s debt, with the income from 4 months of the top 0.1%.
…and that’s just 30% income tax, it used to be 90% for the rich right after WWII: History of taxation in the United States
It’s not that simple, there’s a response of the “top 0.1%” moving their property elsewhere or distributing it by various legal means so that they’d have to pay less.
In dumb terms, you have to design a system where 4 people collectively owning 4bln$ would pay the same as 1 person owning 4bln$. Not even mentioning that they can have N friends abroad.
Also there are still “rich” people in Scandinavian countries, who may not directly own nearly as much as Bill Gates, but still have enormous power.
Also this will, in fact, affect inflation.
My point is - money represents power, which is convertible into other means, you can tax money or property, but you can’t universally tax power.
Money-wise (as a universal equivalent in a non-coercive system) you can at least somewhat clearly evaluate that power. If you scare powerful people off to convert their power into more obscure media, you won’t have that clarity.
So I don’t see this as a problem one can solve, but I see other problems more accessible, like patent\IP\trademark\certification laws.
You’re ruining the circlejerk with your realism! 😠
Edit: I think Mr. Robot gave a good glimpse what would happen if all debts were wiped. It sounds fun on paper, but in the end, the people with the least money would suffer the most.
I personally just lose any interest in conversation when I realize that my counterpart doesn’t want a working system or a better world or really some justice, they simply want to rob someone who has more than they do. No deeper purpose or something, just plain envy.
It’s like certain moments in sex. So bloody frustrating.
And, of course, the only leftists I’ve encountered who wouldn’t be what I describe were book characters. Yeah, nice characters, fascinating, really making me wish something like this was possible, but even with the depth limitations for describing an entire person on paper they were still deeper that RL leftists, FFS!!!
I have at least met living sincere good-willing ancaps and living sincere good-willing fascists (sic) even. The only people I know in person I could possibly call a real sincere good-willing leftist would be my sister, and maybe one of my cousins, and one DM (though from a few conversations I suspect he just has, eh, a leftist background, but is more literate in economics than such people usually are).
Please don’t confuse wealth and money.
What about earnings and income?
Top 0.1% don’t have earnings and income.
I already know I’m gonna be downvoted for this, but the top 1%/0.1% spending isn’t gonna change, whereas the bottom 10% will cause inflation… That’s why there’s no magic bullet.
The bottom 10% don’t have enough money to “cause” inflation, not even the bottom 90% have that much money. Inflation is driven by the top 5-10%, representing 70% of the wealth; the rest just get taken for the ride.
You’re right the top 1%'s spending won’t change, it’s already 1000x above a person’s basic needs, so what’s the difference between 1000x and 900x (10% inflation).
Exactly, the bottom 10% don’t have enough money, meaning that any money you give them will go towards consumption. The top bracket’s spending as % of income or wealth is tiny and is mostly independent of their income. Their money is spent on investments, not basic goods and services. They practically don’t affect inflation.
I think money should be printed during periods of low inflation. E.g. Japan could have benefited from that. After this bout is over, governments can return to printing, carefully.
what steps are being taken to ensure it doesn’t happen again? was any personal data compromised for users?
Good point, I’ll update the post.
Also I am curious, what’s the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?
Mail: info@lemmy.world Mastodon: @mwadmin@mastodon.world Matrix: https://matrix.to/#/#lemmy-support-general:discuss.online
Why wasn’t there an info on /lemmy-world.statuspage.io ?
I think the admins that were on it didn’t think of updating the status page…
Would it be a good idea to have a secondary email not attached to lemmy.world in case of a domain hack?
info@mastodon.world
deleted by creator
So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I’m sure they were only interested in admin cookies, so most others were “useless” to them? I see nothing wrong with my posts so I should be safe, right?
Prior to the JWT secret being rotated, yes, they could have authenticated as you. The tokens are now all invalid and useless
Probably. I had to re-login myself.
If you think they could change your password:
YES, they could.
They could have changed the email => “Forgot PW” and with that you lost ur account.
I think I’ve lost my account, I clicked Forgot Password and nothing came into my mailbox. This account is the one I made just now.
My old account:
If you see that account post or comment on anything, please report it
Edit: Nvm, I use another email to sign up for Lemmy and forgot about it
Report it directly to Ruud or otherwise he will just delete it.
actually nevermind, I forgot that I use a different email for Lemmy, I can log back in now
Nice work on the recovery, especially from a 0-day.
deleted by creator
They defaced it with dicks and changed the federation list to be only threads.net. I don’t think it was a state sponsored chinese hacking group. :)
I’m ok with the dicks but the threads are TOO FAR!!! shuffles off to the angry done**
Thank you all for staying on top of it.
right after the update we also had most of the serverlist cleared except threads.net (which was the last one added so i assumed it was some bug) – otherwise nothing appears to be touched on this instance tho.
Thank you for your work 🙏
First - really good summary and sounds like everyone is working hard.
Cross posting the below comment.
Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.
There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.
If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.
Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.
For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.
It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.
Thanks for the info. We’re looking into this.
Out of curiosity, where would the regulators go for a case like this? There’s no “company” running it per. se.
It seems the general consensus is GDPR applies even to OSS non company entities, but it would appear that there’s very little being done to honor it.
This article outlines Fediverse and responsibilities, I think it mostly requires someone to file a lawsuit before there’s any action.
In another case a man had cameras in his back yard that could also see a public area and was fined and forced to move them.
https://www.termsfeed.com/blog/gdpr-exemptions/
Mainly it just seems to be fodder to be used in lawsuits to make people comply with others security wishes. Not certain how all that works since cities are covered in public cameras.
I am not sure how a platform like this will work with GDPR - each server will be responsible themselves, but how it works with the flow of data between servers and who the regulators would have cases against - I think that is to be tested at some point.
If a valid browser token gets stolen like in this case, then MFA won’t do much good because the stolen token will already have been authenticated. Linus Tech Tips experienced the same thing recently, you can check out their channel.
Can 2FA be enabled for all users? I don’t see the link to activate it after saving.
Just reload the settings page after saving and you’ll see the activation link. Just now enabled 2FA for my account.
Don’t log out! Open private tab and try logging in to test that it works. Lemmy uses SHA-256 TOTP digest which may not work correctly with some authenticators, only generating useless codes.
Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.
EDIT: it has been added back to the block list.
the details of the vulnerability are already known now anyway since there’s a fix that was proposed on the Lemmy GitHub so I don’t think it will hurt others to talk about it
Could you please link the issue? Thanks!
https://github.com/LemmyNet/lemmy-ui/pull/1897/files found it myself
yup that’s the one
what I find weird is that the “fix” still focuses only on the front-end, the issue is still that unescaped HTML is being stored in the database and still trusting the front-end is nuts
I mean, I’m pretty sure that for an XSS attack that’s fine. The entire problem is that somebody posts e.g. a comment that contains code that is automatically run in users’ browsers. If you make the front end just not execute that code then it’s fine. Who cares what’s stored in the back end?
I mean, it would still be better to have multiple fail-safes, and they probably should still sanitize text entering the database.
But this is sufficient for a quick fix.
Let me introduce you to my friend, Little Bobby Tables… :)
ALWAYS SANITISE!
I think people are forgetting that it’s somewhat obvious the hackers or whomever, I don’t really care honestly are Lemmy users considering they did this at night and got into the site so quickly to begin with, they’d have to have been familiar with it to get into it as fast as they did.
If anything everything should be fixed.
For sure it is sufficient for a quick fix. But a Lemmy post can be posted not only on Lemmy but on other front ends (like kbin, mastodon, and many others) and they can suffer from a similar attack due to the backend storing and forwarding the bad content. So, it should not be stored as it is in the backend
I think the main developers are aware of either of them but I’m not sure, haven’t seen anyone site admin wise talk about this mess.
This discussion on the original bug report does talk about the back-end needing a fix as well.
The devs are aware
Oofof… That’s not suppose how we announce vulnerabilities…
Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?
Good point. I did post about this on Mastodon @mwadmin@mastodon.world
Thanks for the reply!
Don’t fall for it. They’re also an admin on mastodon.world! :)
Now I don’t know who to believe! Is Lemmy even real?
Are we all just lemmings? Oh…
Yes.
Well done all involved. Sounds like it was caught and mitigated quickly
IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.
That’s something a hacker would say.
🤣
Good job. I don’t understand very much of that, so that makes me all the more grateful. Thank you.
Can I ask some possibly dumb questions?
- What is JWT?
- Was any private user data compromised, and if so will users be informed?
- Is there anything regular users can do to avoid their data being compromised? For example, not accessing lemmy on certain web browsers?
Thank you!
1 - jwt is the authentication cookie u get when u sign in with ur password, ur browser stores the jwt and then any further interaction authenticates using it
2 - yes, userscripts, extensions, custom frontends, apps, all apps have access to jwt
- JWT stands for JSON Web Token. It’s basically a way for a server (lemmy’s) to put a piece of information in your browser in a way that makes sure it come from the server. It (usually) uses some form of digital signature. You can think of it as a note someone gave you with their signature, assuming said signature is very hard/impossible to counterfeit. The next time you see that person, they don’t have to remember you, they just have to check the signature. If it is valid, anything written on the note is taken at face value.
When you connect to a site, there are a few steps to validate that you are who you say you are (identification and authentication). Something like inputing you login/password. Since it would be tedious to do that on every requests, the first time you give your login/password to the server (this is the simplified version, this exchange is a bit more complex usually) the server gives you that JWT. For every subsequent requests, your browser automatically send that JWT that is simple to handle but hard to counterfeit, and the server safely knows that you’re whoever is written in that JWT.
-
I assume there will be a post here when more details are known, or that this post itself will be updated. As with any online service, it’s up to the service to decide if they want to communicate. (it may also be a legal requirements in some places to tell user when such an event occurs). Since we’re talking about obtaining other user’s authentication token including an admin, it is safe to assume that whatever an admin can see has leaked. This can range from basic user informations to more private stuff, although I am not familiar with the software behind lemmy. Note that this is a worst-case scenario; an admin impersonator could have access to anything an admin could see, it does not mean they immediately dumped everything. It depends on their motivation.
-
Protection against this kind of stuff Compromission of the JWT can happen in many ways and I don’t know which way was used. But if there’s a flaw in the software used (the lemmy’s client-side code, for example) there is not much you can do. JWT can leak through many things :
- server compromission (out of your control)
- client-side compromission (only happens when using a browser; applications that uses API should be less susceptible to that)
- vulnerable extension: if you have browser extensions, they can easily peek into what’s happening in any given page (that’s their whole purpose). Malicious extensions, or extensions that allow outsider’s some kind of control over them can leak data
- browser vulnerability: keep your browser up-to-date, and (this is controversial) stick to a family of well-enough known browser. That obscure browser that have 20 users worldwide and is based on a three years old version of chromium is not the best thing to use
- keep your data safe: only put the minimum required amount of data on any service. For lemmy, I assume an email address and your login/password is the bare minimum (well the email is already extra, but it’s very convenient to have). Some services really likes to get everything they can out of you.
Basically, stay up to date and don’t use shady stuff. Easy to say, I know.
Thanks for your efforts. I know that Lemmy was put in place rather quickly as a Reddit alternative. But I’m genuinely hopeful that this will be a good alternative.
How does this impact those using mobile apps like Jerboa or Liftoff, instead of the website directly?
Check the pinned post on liftoff community page.
https://lemmy.world/post/1292303
As a safety precaution logged-on sessions on many servers have been cancelled and you are required to logon again.
Thanks, I’ll do that. Curiously, the lemmy.ml account keeps working, wonder what it depends on.
Probably cause they haven’t cancelled the session
Was wondering this myself. Is there a way for users who where exposed to know about it?
(Edit) Eg if the exploit was through a post get notified if they saw the post?
apparently they posted it as a weird image or emoji that looked like this:
HOLY SHIT I LUCKED OUT LMAO, I ALMOST CLICKED ON THAT EARLIER
There is no need to get notified, they didn’t steal passwords, just session cookies. Most (all?) servers have invalidated all the user login cookies, but if you are in doubt, just logging out and back in should be enough to get a new cookie.
as someone who uses the app, extremely little effect from my experience, I didn’t notice something was wrong at all until people pointed it out due to how liftoff does the whole sidebar thing for the instance.
It’s still better to change your account password and clear your cache.
One thing I don’t get. Custom emojis can only be created by an admin, but you’re saying an admin’s account here got compromised because of that and not the other way around. Does that mean that an evil instance set a custom emoji with the injected JavaScript and propagated it to the federated instances?
From the fix, I believe the custom emojis were not double checked after a user submits a post. The post data was used to display the emojis, and thus allowing injection.
The fix now is to search the emojis in the custom emojis list from the backend rather than the user post.
