WC Archive

Wow, scary stuff...

Jamais, I agree with most of the points in your fine piece, and I generally think knowledge is better than ignorance. Still, the title "Safety In Knowledge" has a slogan-like ring to it. There's no automatic correlation.

Knowledge is a tool. Like any tool, it can be wielded constructively or destructively. Substitute another tool in your title: "Safety In Chain Saws", "Safety In Pulleys", "Safety In Cuisinarts". Well, it all depends. The underlying ethics, codes of conduct, legal agreements, transparency, accountability and consequences of society guide what we do with knowledge. Rules govern tools. Tragically, the rules are often bad.

Perhaps there's a comforting reason why the Armed Forces are involved in this research? Perhaps it's all guided by benevolence and is completely transparent and accountable? Boy, I sure hope so.

Publishing the Genome to this horrible virus was an unnecessary risk. Even some Biologists are questioning the wisdom of doing such a thing. Kurzweil(or whatever his name is the guy who predicted the singularity in 30 years) said this is akin to posted the full plans for a nuclear bomb on the web except 10x times more dangerous as it doesn't really need any exotic materials like fissinable fuel.

Even in the Sciences Human Stupidity knows no bounds.

An sars didn't kill thousands of people it killed slightly over a thousand according to the Red Cross.

Security through Obscurity Security through Infamy

Maybe?

Sorry, poor formatting in earlier post.

You seem to be challenging the doctrine of
Security through Obscurity

Perhaps you should use the arguments of
Security through Infamy

Although it is questionable whether commenting entities such as myself are encouraged to challenge the title of essays that we already agree with.

You'd figure Kurzweil and Joy, both being computer scientists, would know that security by obscurity entails risks of its own. In cryptography the ciphers that get trusted are the ones based on open and published algorithms. How can we know they're safe if we can't test them fully ourselves? Many network security specialists trust systems based on open source more than proprietary systems. I this applies here.

Secrecy doesn't solve the problem. It only drives it out of public sight into military labs, terrorist weapons deals and well-educated but poor weapons scientists.

Have we learned nothing from five decades of Cold War? The United States thought it could keep the secret of nuclear weapons safe for decades, the Soviets disproved that in less than 4 years. I'm sure the Soviets thought their rockets would be their secret forever but only a year or so after Sputnik, the US launched its own satellite. MIRVs, stealth, supercavitation, phased-array radar, the list goes on and on.

Military secrets are the most fleeting of all secrets.

What we need is more openness, trust building and public inquiry not more secrecy and paranoia. Secrecy drives arms races. Regulation of dangerous knowledge works best under wide public scrutiny. I agree with Dave and Ben, this is scary stuff but censorship will hurt us more than it will help us.

And if we've been reading the news for the last few years, we saw this day coming. Remember back in 2002 when they synthesized the polio virus from scratch? Only few years before they were still debating whether to destroy the sole remaining smallpox viruses from the US bioweapons stockpile. It looks now like it wouldn't have really mattered.

I say give the world's scientists all the information they need to defend us against these threats.

Let's not forget serenip either. Perhaps study of the 1918 flu genome might teach us something unexpected that will help us defeat HIV that much more quickly.

this is not a case of security through obscurity. see http://en.wikipedia.org/wiki/Security_through_obscurity.

kurzweil and joy were not suggesting that the "algorithm" be hidden, for instance not saying where it is the secure area(s) the research is being conducted. they were saying that the DNA sequence, which is the equivalent of the cryptographic key, be hidden.

the line of thinking i'm reading here seems to suggest that all secrecy is bad. am i to understand that i should publish all my passwords? what is the criteria for secrecy?

I stand corrected.

And I didn't mean to give the impression that we should pass out our car keys, passwords and credit cards to random people on the street. Obviously accesss to some kinds of knowledge show be restricted. But we should never assume that this alone keeps us safe.

I was attempting to say that sometimes secrecy can be self-defeating. Secrecy drives fear and makes it hard to find abuses. Banning scientific knowledge only drives it overseas or into poorly monitored military research projects.

And now I just realized another flaw in my vague analogy. The 1918 virus genenome isn't really like a cryptographic key because it doesn't really lock anything up. It's not a password.

The only way the analogy holds up is in terms of cracking. Once someone demonstrates it can be done, that the genome can be reconstruted, it only gets easier and cheaper. Censoring the genome only slows this process down by scant years. Censoring the genome doesn't seem like a very good defense to me.

I just finished reading the NYT editorial. My mind boggles: not only are these computer scientists - how of all people should know better - arguing for secrecy, they're calling for a biological Manhattan project. What a poor choice!

There is little chance of international agreement on what to do with viral genomes. If one player spends billions presumably on "defensive measures" and doesn't publish all the data, the result of any negotiation is 100% predictable.

On a more serious note, people should watch this videos of Jon Stewart about the Bird Flu: http://www.boingboing.net/2005/10/17/daily_show_on_bird_f.html

Yes, MORE serious. Security through obscurity is sheer nonsense. Laugther however is proven to be good medicine :)

Thoughtful comments - thanks everyone. Ben Hunt, thanks for (obliquely) pointing out that commenting on titles may be off base. I think I was being oblique too: I meant to ask whether all knowledge should be widely known, and whether handing out tools to everyone guarantees that they'll build something positive.

I've thought more about this particular case, and I agree with Jamais. Research is often done by "tribes", each often jealously guarding their "ownership" of vital knowledge. One can recall the egos, pettiness and spite tainting some of the first efforts to understand HIV/AIDS. Cooperation, sharing progress and results - understanding that solving the problem is more important than being right - is better. When the next flu pandemic hits, it will overwhelm the capacity of any one nation or group, so we might as well get better at cooperating.

The concerns of Kurzweil and Joy are valid, but, as Anne Herbert once wrote, ultimately the only secret that matters is that something is possible.

Why do technophiles always disrespect those who tell them they can't play with their tech toys. I saw everybody attack Bill Joy for being naive in 1999 at Stanford when he debated Kurzweil on this topic and now they attack him and Kurzweil.

I think technologists are biased towards openness of information and ofer many arguments much as a child cries when their parents tell them to stop playing with their toys and go to sleep.

If Kurzweil and Joy agree on such a serious topic people should listen seriously, yet all i see are tech threads where people (usually young tech people) are trying to debate them.

Open source the genome!!!!! Yeah! Only problem their is that YOU are the copyright holder of your own genome!

First, about the number killed by SARS: I've changed the entry to read "thousands infected," which is a more accurate phrasing. Estimates of the number killed vary by the source, so going with the number infected (which most sites say ranged from 8,000 to 9,000) is better.

I don't really agree with any of the cryptographic analogies presented here, but I do agree with Pace's original statement that Kurzweil and Joy advocate security through obscurity. It's a pretty classic argument of the type, actually; the core of their argument is that we'd be safer if the sequence information was kept secret. Unfortunately, that's only true if you ignore the bigger picture, the variety of issues I touch on in the post.

The logic here is pretty straightforward:

* The 1918 virus and the H5N1 virus are broadly similar, with the biggest difference being that 1918 had mutated to be human-transmissible.
* We know from recent experience that global collaboration using open-access information can mount a very effect response to potential pandemic viruses.
* Studying the way that 1918 achieved its virulence is likely to give us a head-start on responding to a human-transmissible version of H5N1.
* The threat from H5N1 is much more plausible than the threat from someone recreating the 1918 virus using its genome sequence.

Bill Joy likes to refer to "knowledge-enabled weapons of mass destruction," and would no doubt include a recreated 1918 virus in that category. But he ignores a couple of important elements: more often, the enabler for mass destruction is nature, not knowledge; and our defenses against such "weapons," natural or otherwise, is also knowledge-dependent. Hiding information like this hurts the defenders more than it hurts the attackers.

Erik, your post (which popped up while I was writing this response) makes no sense. This has nothing to do with "technologists" having their "toys" taken away; this is about medical scientists around the world having access to information that would help them save lives. And it has nothing to do with the human genome, open sourced or otherwise. I'm really not sure what you're trying to say, other than that you disagree.

I saw so much anger against Bill Joy for the last 5 years for even ASKING the question what should we do about technology. People made all sorts of personal atacks on him (I am not including you here Jamais) and I began to notice how people who work in tech and made their money there had a bias towards progress and a almost paranoid concern that any type of "control" would stop innovation in its tracks. Many techies see only the short term effects of any control but fail to see that in the big picture innovation will still happen exponentially.

Innovation today is happening at an exponential rate as many of us understand. I believe that having some controls on some data will have little overall effect on the rate of innovation but may enable us to direct away from very dangerous places and towards rewarding ones. Techies seem to think that we even could make the genome secret. The train is coming down the hill and there is no stopping it, but we still may be able to steer it a bit.

The genome is where computer science was at in the early 1980's. At that point if we would have paid a bit more attention to security there would still have been innovation but there might be more security built into the networks. Many computer people would say that the computer virus writers do a favor by pointing out where systems need to be built up. In the real world biohackers can kill millions and destroy societies. We must do everything we can to prevent biohackers in the first place. The consequences are many, many orders of magnitude more severe.

Genomic research needs to be increasingly controlled. The defenses will and should be done by serious labs in private industry, universities and gov labs where there is oversight and resources. Garage and individual labs, while still contributing to the computing world, have no place in the genome. Between USA, Europe, China, India, Singapore, Japan, Russia there are MILLIONS of scientists working on the genome and they will all have qualified access. In 1918 the world was a very different place. Even institutional research was crude.

Music, videos, linux code can all be open sourced. But as owners of our own genomes, we should al be concerned about the lack of controls on this genomic information. If the genomic world becomes like the computer world we may all die.

I make these analogies because I think that computing people and techies have alot of opinions about the genome. They take their experience in computing and tech and extend them to the genome. I have worked in the biotech field for 20 years and feel the analogies have serious limitations. The genome must be seriously respected. Hacks can be massively deadly. Even good natured hacks should be tightly checked.

There should be massive governemental effort to direct efforts. Unfortunately now for example, the private sector has little impetus to find vaccines as the main customer is the poorly paying governement and even they may steal the patent in an emergency. The main success there is for private research in rapid diagnosis is Cepheid.com and they have not been rewarded in the financial markets (Yet). But cepheid came out of Lawrence livermore labs and there hs been no garage science there, thats all gov and Darpa funding.

The constant comparisons between nuclear and biological weapons in the media surprise me somewhat, for the simple reason that nuclear weapons are at least "localized." These are very different threats.

A pandemic knows no political boundaries, and while terrorist actions show a total lack of regard for human life regardless of creed, there is a distinction between terrorism and, well, nihilism. Terrorist attacks are designed to send messages about geographic or at least demographic distinctions. For this reason a nuclear attack seems to me to be a far more likely threat from terrorists than a biological one.

That said, while I agree entirely with what Jamais is saying and disagree wholeheartedly with Kurzweil and Joy's argument, carrying the principle that all knowledge should be free to extremes has sobering consequences: does dedication to the ideal that open knowledge improves security also imply that one must support putting plans for nuclear weapons on the internet?

Is it even feasible that disseminating a "Nuke HOWTO" could improve the security of the world?

Personally I think that would be nuts, but it's important to think about where the line should be drawn.

Patrick: there's a couple problems with that reasonning. One is that the how-to's are almost certainly already in the hands of the people that would want them. A dirty bomb is probably easier to make too.

More to the point, the nuke HOW-TO doesn't make defending against a nuclear bomb any easier. Publishing the genome does allow numerous researchers the possibility of looking at the mechanism that produces disease, and create effective counters.

It's obvious that this could be worse than a nuclear weapon, and you're absolutely correct that this won't respect national boundaries. This in turn means that any power that would want to use it for aggression would need stockpiles of medicine, unless they really didn't care about their population or the economic impact of a mass die-out. God, I feel so strange thinking about this so rationally... maybe how people felt looking at the Cold War's MAD. Mutually assured infection?

the cost of sequencing a genome is dropping every year and thus presumably is it's accessibility to anybody for anyreason. Just guessing, It seems reasonable to imagine someone with small amount of ingenuity could have access to any viral infection out in the open and sequence it for less than $500K today and less tommorrow. Perhaps it involves breaking it into pieces and sending to different labs. So it's as hard as figuring out one is building a bomb or just stocking a farm from just looking at the ingredients.

As to defenses it's game of velocity of innovation. Publishing presumes that more people will be watching, and working on something a reasonably safe assumption. Collectively we do have a massive mount of computational ability, but it's possible that infections can spread faster than our ability to contain and react, if it's detection is late. I think this is especially true given viruses ability to mutate. So instead of fighting one virus the system is taxed fighting 6 varients and growing. Also think of stock markets, they crash all the time taking peoples investments, despite peoples best strategies over years at coming up with protections as most protection assume a shallow dip in the market instead of a catastropic event.

Most of our defenses rely on containment, and with our vastly connected world propgation is surprisingly fast. Sars may have been well contained or just not virulent enough. But how does one effectively stop animal/insects from flying and spreading. West Nile Virus is already present in most mosquito populations.

My guess is that there are less than about a 1000 labs worldwide that need to be working on vaccines. In reality there are probably many thousands, but there does not need to be. Take Crucell.com for example. They have some very good tech for making recombinant vaccines. There does not really need to be thousands of these. A few hundred companies with great Manhattan project like funding could accomplish alot while still not putting masive amounts of genomic data into the net. I doubt a overworked grad student or a thousand could innovate a vaccine like Crucell could, either because they do not have the advanced technology or the funding. I also would not want to give a thousand grad students Crucell's technolgy.

You could have 100,000 miscellaneous labs backed up with 50 Billion dollars and lots of open source info OR you could have 1000 mega-labs also backed by 50 Billion and many controls on data export. I would argue that the risk would lower of biohacking while the ability to innovate would still be strong.

Remember, much innovation is patent protected anyways, so unless individual researchers were infringing on patents they would not be doing cutting edge work that is likely to get things done anyways.

As far as fear mongering about the 1918 flu goes. I think there should be more! I was at a conference in 1997 where a scientist said that he visited a lab where, just as an experiment, inserted antibiotic resistance to every known antibody. Took one week, they quit when they started thinking how scary that was. 1997! In the late 1970's (?) Russia designed Smallpox that was 1000X more virulent and simultaneously more resistant to treatments. They made enough to kill everybody on Earth. Also in 1997 I heard a news report that a pair of Israeli researchers (grad students?) were caught working on genes specific to Arabs. They said it was to help find cures for diseases of Arabs. Of course what can specificaly help can kill or genocide which the Israelis are very aware of and sensitive to.

I would say that the potential reality of designed pathogens is beyond what even your scariest fear mongering could imagine. An atomic bomb might kill 5 million people in our most crowded cities. In 1918 50 million died. That already made it worse! It is actually a dillema; the more people talk about it and get concerned, the more possible it is that something happens. That is why in true Manhattan project style, a world wide well funded semi quiet project should take place that tries to develop possible treatments for these problems ahead of time. This should have strong gov and UN oversight, have strong academic and private industry participation.

If someone takes the viral genome, they might also add extra virulence factors maing it into massively lethal. When 911 happened many people said " Who could have imagined?" Well I think many people did but were not taken seriosly. Some naturally occuring pathogens can be very dangerous but it really isn't the naturally occuring ones we really need to worry about (even though they can be terrible).

Having said all this I doubt a Manhattan project will happen. Other countries probably do not trust us enough and George Bush chose NOT to sign the Biological weapons non proliferation treaty soooooooooo......