I’m finding it harder and harder to tell whether an image has been generated or not (the main giveaways are disappearing). This is probably going to become a big problem in like half a year’s time. Does anyone know of any proof of legitimacy projects that are gaining traction? I can imagine news orgs being the first to be hit by this problem. Are they working on anything?
I actually spent a few years of my life writing a whole software project for exactly this purpose, and I still think that it’s the only practical way to solve this problem.
Called “Aletheia”, it ditches the idea that software can spot a fake entirely and instead provides a way to guarantee the author of a piece of media. If you trust the source, you can trust the image/video/document/whatever.
If you’re curious, here are a few relevant links:
If you trust the source, you can trust the image/video/document/whatever.
I think this is key. There have always been sources that have been incredibly trustworthy that people choose to ignore. Like AP News or Reuters for example. Sometimes they might make mistakes but as long as they keep fixing them in a timely manner that’s what’s important.
Yeah it’s still possible they could post an AI image - but that’s why you compare sources.
Besides that would ruin the credibility they’ve spent decades building.
Read the beginning of the white paper two - great work! I can tell you put care into this.
Is this similar to how the NFT was supposed to work?
Thanks! And no, this is absolutely nothing like NFTs.
NFTs require the existence of a blockchain and are basically a way of encoding a record of “ownership” on that chain:
Alice owns this: https://something.ca/...If the image at that URL changes (this is called a rug pull) or a competing blockchain is developed, then the NFT is meaningless. The biggest problem though is the wasted effort in maintaining that blockchain. It’s a crazy amount of resources wasted just to establish the origin.
Aletheia is much simpler: your private key is yours and lives on your computer, and your public key lives in DNS or on your website at a given URL. The images, videos, documents, etc. are all tagged with metadata that provides (a) the origin of the public key (that DNS record or your website) and a cryptographic proof that this file was signed by whomever owns the corresponding private key. This ties the file to the origin domain/site, effectively tying it to the reputation of the owners of that site.
The big benefit to this is that it can operate entirely offline once the public keys are fetched. So you could validate 1 million JPEG images in a few minutes, since once you fetch the public key, everything is happening locally.
Thanks for the explanation!
Sounds much more similar to something like NOSTR. I’ve always like the simplicity of using a keypair as an identity.
Does anyone know of any proof of legitimacy projects that are gaining traction? I can imagine news orgs being the first to be hit by this problem. Are they working on anything?
In short, no. There’s no solution to this crisis of legitimacy and no way to validate that a work came from human hands that can’t be counterfeited. Short of a Butlerian Jihad, it won’t be going away either. I’m looking forward to the inevitable bubble pop that will kill the overinvestment in generative tech but the models will still be around even without big companies wasting billions to train new ones.
In short, it’s long past time to start treating every day like April Fools Day.
Butlerian Jihad it is then!
deleted by creator
Time lapses of drawing the image would be near impossible to replicate currently.
Only a matter of time and they’re already good enough to fool the gullible.
My understanding is that the generator would have to make each frame from scratch and also keep track of the progress of the drawing.
They may have trained on a few Timelapse drawings, but that dataset is much smaller then the photographs and artworks in the models.
I’m sure it could happen, but I’m not sure there will be enough demand to bother.
Universal Basic Income is really the only answer. So we can make art for fun instead of as a means to survive.
The problem is Goodhart’s Law; “Every measure which becomes a target becomes a bad measure”. Implementing a verification system that depends on video evidence creates both an incentive to forge such videos and a set of labeled training data that grows more readily available as the system sees more use. The adversarial generative network is literally designed to evolve better scams in response to a similarly-evolving scam detector, there’s no computational way around the need to have people involved to make sure things are what they’re claimed to be.
Universal Basic Income would be a good start, but the fundamental problem is money as the primary organizing force of society.
The final output image is just a grid of pixels, just like any other image. Assuming the image has no metadata or has been stripped of metadata, how do you really tell the difference in the first place?
Sure, you can look for JPEG artifacts and chromatic noise and all, but it’s pretty easy to overlay that on top of an AI generated image to make it appear more legitimate at a passing glance.
I really don’t know a good answer to your question right now, but I’m definitely interested in whatever useful answers others might offer…
You should never have trusted images before generative AI either. Trace the source and only trust the image if the source is legitimate.
Negative proof: the AI company signs it with their watermark.
Positive proof: the photographer signs it with their personal key, providing a way to contact them. Sure, it could be a fake identity, but you can attempt to verify and conclude that.
Cumulative positive and negative proof: on top of the photographer, news organizations add their signatures and remarks (e.g. BBC: “we know and trust this person”, Guardian: “we verified the scene”, Reuters: “we tried to verify this photo, but the person could not be contacted”).
The photo, in the end, would not be just a bitmap, but a container file containing the bitmap (possibly with a steganographically embedded watermark) and various signatures granting or withdrawing trust.
Isn’t that more like trusting your source though, which media companies either do or don’t do already.
It would be a method of representing trust or distrust in a structured way that’s automatically accessible to the end user.
The user could right-click an image, pick “check trust” from a menu, and be presented with a list of metainfo to see who has originally signed it, and what various parties have concluded about it.
Perhaps a trusted certificate system (similar to https) might work for proving legitimacy?
Certificates like that can only guarantee that the work was published by someone who is the person they claim to be, it can’t verify how that content came to be in their possession.
Hmm, I see. Surely wouldn’t that be enough if they proclaimed they would only sign real photographs though?
Anyone can make such a promise. Verifying that they have followed through with it is not a technical challenge, it’s a socioeconomic issue.
https://contentauthenticity.org/how-it-works
The page is very light on technical detail, but I think this is a system like trusted platform modules (TPMs), where there is a hardware root of trust in the camera holding the private key of an attestation certificate signed by the manufacturer at the time of manufacture, and it signs the pictures it takes. The consortium is eager for people to take this up (“open-source software!”) and support showing and appending to provenance data in their software. The more people do so, the more valuable the special content-authenticating cameras become.
But TPMs on PCs have not been without vulnerabilities. I seem to recall that some manufacturers used a default or example private key for their CA certificates, or something. Vulnerabilities in the firmware of a content-authenticating camera could be used to jailbreak it and make it sign arbitrary pictures. And, unless the CAI is so completely successful that every cell phone authenticates its pictures (which means we all pay rent to the C2PA), some of the most important images will always be unauthenticated under this scheme.
And the entire scheme of trusted computing relies on wresting ultimate control of a computing device from its owner. That’s how other parties can trust the device without trusting the user. It can be guaranteed that there are things the device will not do, even if the user wants it to. This extends the dominance of existing power structures down into the every-day use of the device. What is not permitted, the device will make impossible. And governments may compel the manufacturer to do one thing or another. See “The coming war on general computation,” Cory Doctorow, 28c3.
What if your camera refused to take any pictures as long as it’s located in Gaza? Or what if spies inserted code into a compulsory firmware update that would cause a camera with a certain serial number to recognize certain faces and edit those people out of pictures that it takes, before it signs them as being super-authentic?
Camera companies have been working on this. They have been trying to create a system that makes it possible to detect if an image has been tampered with https://www.lifewire.com/camera-makers-authentication-prevent-deepfakes-8422784
However this signature probably just uses assymetric encryption which could mean that the signing key on the device could be extracted and abused.
Being able to “prove” that something is AI generated usually means that:
A) The model the generated it leaves a watermark, either visually or hidden
B) The model is well known enough that you can deduce a pattern and reference what you’re checking with that pattern.
The problem with the former is that you are trusted this corporations (or individuals training their own models, or malicious actors) to do so.
There are also problems with the latter: The models are constantly iterating and being patched to fix issues that people notice (favoring certain words, not being able to draw glasses of liquids full to the brim, etc)
Also, if the image or work was made using a niche or not well-documented AI then it probably wouldn’t be a pattern that you’re checking.
Also also, theres a high false positive rate, because it’s just pattern matching mostly.
Can’t.
Adversarial efforts are how some networks get trained.
Yes, several large AI companies themselves are “watermarking” their images. https://www.nytimes.com/2024/02/08/business/media/google-ai.html
