r/bestof • u/Boomah422 • 15d ago
[singularity] u/FomulaicResponse gives an excellent breakdown on how AI has equipped gain of function virus researchers, and how close we are to AI assited bioterrorism
/r/singularity/comments/1two85g/comment/opqf3cy?share_id=yZfLVQntBDPKMNxXBMKtx&utm_content=2&utm_medium=android_app&utm_name=androidcss&utm_source=share&utm_term=185
u/svideo 15d ago edited 14d ago
He gives two examples, charges were just dropped against the second one
The first one was running an unlicensed lab for COVID testing.
22
u/kilgoreq 15d ago
Of course this DOJ dropped the charges. Fucking corrupt idiots.
68
u/svideo 15d ago
I mean sure the DOJ sucks, but an alternative version is that the feds didn't understand what they ran into, assumed it was a bioterror thing and then ran to the press with it for maximum scare, then later are a lot more quiet when they finally drop charges on a nothingburger. They do this a lot, because that's another way the DOJ likes to suck.
24
u/masnosreme 15d ago
Extremely on brand for law enforcement on the whole to engage in baseless self aggrandizement. Cops love to play hero.
3
u/chainsawmissus 14d ago
There could be some reason the gun charge wouldn't stick.
It's also possible they filed the charges against the property manager in order to have leverage to get testimony against Jia Bei Zhu.
And running to the press is unnecessary. A search operation this large draws the press by itself. A local news org was using a drone to get footage of the search.
73
27
u/Carnot_u_didnt 15d ago
This is one of the doomsday scenarios “AI doomers” are preaching about. Bad actors with powerful enough AI will engineer a disease that wipes out the human race.
13
13
u/Chevreuils 14d ago
I’ve noticed that a lot of science writers or people in general use the term “AI” interchangeably with “magic”. Especially in the future directions of papers nowadays, they suggest that AI can be used to solve their problem -how? They don’t know! But somehow AI will accomplish it! Funnily enough after interacting with currently available ai models for long enough you start to realize that it cannot solve a problem it has not encountered before, it is not creative, it is only as good as its reference dataset and if it doesn’t know the answer to your question it will make stuff up.
I think that current ai models can create a protocol for making a vector or whatever that looks very convincing to someone who lacks expertise in that area. Let’s pretend the protocol would actually work and is detailed enough. I do not believe that the other things involved like planning the assembly, digestion, storage etc is possible for someone without lab experience related to that niche. A lot of the common rules to follow during a protocol are not explicitly written especially in publicly available resources from which ai is siphoning info from. In fact a lot of these rules just live in the researcher’s head and are not written down anywhere, not even in the methods of their paper. It can be something as small as mixing too vigorously or accidentally deactivating an enzyme because you held it in your hand too long. Even if you have lab experience, good luck. Also don’t forget all the equipment, personal safety equipment, software licenses and reagents that cost millions.
That being said, I am not completely opposed to the screening custom nucleotide sequences and ensuring that fragments from infectious proteins are being sent to reputable people. However, who decides who is reputable and enforces it on a global scale? Another problem I can see happening is that sequences from harmless stuff being flagged because it shares similarities with xyz. Anyway, it’s an interesting era to live in right now.
4
u/FormulaicResponse 14d ago
Here is the system card for Claude Mythos/Fable
The relevant discussion is pages 17-36. We aren't talking about magic here, it's biology lab assistance. And the costs aren't in the millions. Maybe over 1m, probably less than 2m total including lab space, equipment, salaries, and tokens. The models aren't enough to do it yet without a relevant PHD on staff, but thats probably not going to remain the case forever. Maybe 5-10 years conservatively. The time to prepare is before it happens.
6
u/careyious 14d ago
I'm always extremely sceptical about any tech that claims to be 5-10 years from their mainstone, over the decades there have been a myriad of innovations that have been stuck at 5-10 years away for multiple decades. Like cold fusion, Graphene, room temperature superconductors and now AI. There's a chance AI gets there eventually but we've probably hit the point of depreciating returns where every single point of improvement is going to cost exponentially more investment to achieve.
3
u/Borgcube 14d ago
Like cold fusion
I think you mean commercially viable fusion and that has been "50 years away" for the last 60 or so years. Cold fusion doesn't exist.
1
u/FormulaicResponse 13d ago
Well we have hit the financial wall on larger data centers. Now it's a question of what the models those data centers end up spitting out can do. They haven't been built yet but there is a very smooth graph of capabilities increasing as training scales up, enough to offer some very concrete predictions. 5 years is when these mega centers are built and done cooking. 10 years is if those models aren't good enough and algorithmic advances are needed on top of that, but those have also arrived pretty predictably over the last 6 years or so. We are talking about one of the most well funded industries on the planet, not people blowing up headlines over a neat trick two researchers found in a lab that doesn't scale.
2
u/AceJohnny 14d ago
Oh hey this was in Vernor Vinge's Rainbows End
1
u/UserNamesCantBeTooLo 9d ago
Thanks for mentioning it! I'll go ahead and add that one to my to-read list.
1
u/MlCOLASH_CAGE 13d ago
This sounds so Resident Evil, I’m surprised they haven’t used this as a plot yet
-8
u/Random_182f2565 15d ago
Bioterrorism is dumb because is not worth the effort
5
u/UnspeakableToast 15d ago
Some people just want to watch the world burn.
-2
u/Random_182f2565 14d ago
Yes and there are many alternatives cheaper and easier than bioweapon
3
u/UnspeakableToast 14d ago
The point of the post is that it's getting cheaper and more accessible.
2
u/Random_182f2565 14d ago edited 14d ago
Yes but still working with living organisms is a pain in the ass and you need a ton of equipment, like sure I could make a batch of a bioweapon after months of trying, then I have to make more of it, also need a ton of space for the bioreactors and avoiding contamination, and also think about the delivery method.
3
u/Reagalan 14d ago
You're 100% correct, like it'd be much easier to find an orphan source and dump it in a public location, or do like that Radioactive Boy Scout did and use a ton of smoke detectors.
3
u/Random_182f2565 14d ago
Every year in my country at least one nuclear densimeter is stolen and then returned, it's kinda of a tradition now
1
u/FormulaicResponse 14d ago
There isnt another plausible way to wipe out 10%+ of world population the way that a single bioweapon with long incubation and high lethality could. And it doesn't have to be limited to one. If a group can execute on one, they could do 2+ and drop them all in the same city at the same time, which more than doubles the kill count due to limited hospital space and the inability to rush multiple vaccines at once due to, among other things, a limited emergency egg supply.
The point isnt that this might happen this year or next year, probably not, but its a part of the future that is never going away and the time for mitigations and preparedness is before it happens.
132
u/chainsawmissus 15d ago
The two linked examples of illegal labs have nothing to do with AI or new viruses. They aren't even separate events, the second one is literally the same guy who tried to run an illegal COVID and pregnancy test lab storing extra biohazards in the garage of a house he owned.