If you are accessing this from the Web, not all links will work. The best way to get to this Flog is via Freenet. Apologies for the 2 updates with no new content, I had to fix some links.

Resources page

2008/02/29

Chat and other matters

Freenet 0.7 is coming along nicely, although the tentative feature freeze is by no means absolute. Ian wants to release in March; a release candidate towards the end of March seems feasible at this point. ULPRs and related code (the cooldown queue) are pretty much finished, and there is a feature freeze, although some HTL changes have been necessary and one more may be in the next few days. Bugfixing is in full swing; the spider and the transport layer have been recent targets. We still have way too many timeouts, but many nodes in one VM simulations (freenet.node.simulator) have helped to find the cause of some...

However, one big piece isn't ready yet: a working, user friendly, hard to spam, bundleable chat client. The spammer seems to have taken a day off today, but until today he has relentlessly spammed all the default boards and many more popular boards, making them unusable. IMHO Freenet is a community, and for any community to function there must be a usable means for chat. Right now FMS is apparently working and under heavy development, but in C, and using an NNTP interface; seagull is apparently working on porting it to java, but we haven't seen his work yet. Ian is of the view that Frost and FMS are separate projects to the node and we shouldn't wait for them. What do you think? Answers to an appropriate forum, should we:

1. Unbundle Frost. We could just not ship Frost. This has the advantage that we could meet our deadline for shipping 0.7.0 more easily. And we wouldn't be shipping any known-vulnerable or known-broken software. But IMHO it would reduce user retention dramatically, at least by 50%.
2. Ship Frost as-is. If Frost isn't being actively DoS'ed, we could just hope that the spammer is a friend (e.g. an FMS advocate), and ship Frost anyway. If he isn't, he will DoS Frost as soon as we ship 0.7.0, even if he takes a break in the meantime. IMHO it would probably be a good idea to mention the fact that Frost has been actively DoS'ed and remains vulnerable in the 0.7.0 announcement: this will put off users, but if they have to discover it themselves, that will put off even more users. Our problem is not getting people to install Freenet, it's user retention: we have huge "join-leave churn".
3. Make it a release blocker and act accordingly. Review third party code, help with porting FMS to java if necessary, ship 0.7.0 with a bundled, working java port of FMS, probably with a web interface based on Worst. Have an official plugin (with anonymous contributors such as seagull, but also with non-anonymous contributors such as me) implementing the web of trust and exposing it via FCP, and another plugin (or an option in same one) implementing a web interface. You can get version 0.3 of the Worst source code here (if you have a more recent version then please post it to fms or to the fms board).

Note that because FMS is in C and therefore not bundleable, I don't currently run FMS. I may run the java port of FMS when it becomes available - hurry up seagull! I've seen test posts, so it can't be too far away...

In other news, we should seriously consider whether to take part in Google Summer of Code this year. I was approached on Wednesday by an enthusiastic and apparently competent would-be SoC student, who now has an SVN account and is working on a bug in the transport layer. Last year SoC was a lot of effort and a partial success - a lot of the code was of poor quality, but some of the devs have reappeared or not disappeared. If we do do SoC this year we will probably take fewer students, and we'll certainly want to be more careful in selecting the student (and not the proposal; proposals can be renegotiated).

2007/10/26

Radical update

A standard criticism of the Christian gospel is that the various resurrection accounts (bibles on freenet here or on the web here) are inconsistent. And at first glance they disagree on many important matters. I just found this gem (from the web), which makes a persuasive case that they can be pieced together into a coherent whole. Why does this matter? Well, if Jesus did rise from the dead, he probably is the Son of God. If he didn't, then Christianity (with all of its widely acknowledged moral contribution to the world) is nonsense. You can of course claim that the whole thing was made up, so it doesn't matter that they are consistent; but they were clearly written by different authors, and reasonably early although probably not direct eyewitness accounts; and Matthew's gospel was written to jews so claiming that the guards' tale was widely known among the jews probably would have been a bad idea if it wasn't true. Have a look!

I can't write a flog entry without a Freenet-related update. And I should probably explain what's going on locally too. Firstly, emu (the freenet project web/mail/etc server) has been down for a few hours, it will be back up soon, due to its being moved from London to Manchester to cut costs for bytemark; we're getting a free memory upgrade in the process. Secondly, we (family) are moving to Scarborough (wikipedia) on Thursday and may not be around for a while after (hopefully we will have a phone line when we arrive, so we should have dial-up, but there will obviously be lots of stuff going on e.g. unpacking). Reasons for the move include health (Bristol's air is really unhealthy), and closeness to relatives, and a much bigger house. Working from home has certain advantages!

Now for developments in Freenet itself: Build 1069 features a major security fix for our connection setup code: It was possible for an attacker to do a man-in-the-middle attack using weak diffie-hellman keys. Tor had a similar vulnerability in 2005, and Freenet 0.5 still has this problem (we are not going to fix it, we will apply a patch if you send in one, but Freenet 0.5 is unmaintained and unsupported). 1069 is mandatory on Tuesday, so upgrade! Hopefully the auto-updater will upgrade your node automatically, or is even now awaiting your clicking the "Update Now" button.

In other news, slow progress continues to be made towards full opennet. Path folding is working, UP&P (automatic port forwarding and IP detection if your router supports it) is working, reconnecting after reasonable downtime is working (I dropped my darknet peers, shut down my node for 12 hours, and it got back on its feet with opennet soon after starting back up; admittedly I'm not NATed...). 1070 will have a fairly major security fix relating to path folding (traffic analysis related to packet size ... sorry), and the major items remaining before we can implement automatic bootstrapping are automatic detection of port forwards (to get rid of the annoying connectivity messages and automatically detect if you are eligible to be a seednode), and anonymous connect crypto (the current link setup assumes both sides already know the other, which is great for darknet, and even for opennet, but not for bootstrapping opennet). So hopefully well before Christmas we will have full opennet support - full meaning you don't need to exchange noderefs with strangers, ever, even when you first connect to Freenet.

That's no excuse for running Freenet in its most insecure mode though! If you know somebody who has a freenet node, it is a really good idea to connect to them rather than to a total stranger chosen by the opennet logic. For several reasons: Firstly, your friend is likely to be more trustworthy than a random stranger, and Freenet is not perfect: those you are connected to can try to analyse your requests (see also the wiki). Secondly, it's not a random stranger: an attacker can choose to be connected to you relatively easily. Thirdly, your node is not invisible, an opennet node can be detected relatively easily, and then blocked on e.g. the Golden Shield (many western nations have some form of ISP-level blocking even now). And it is likely that one day Freenet will be illegal, even where you live, whether by the IPRED2 (pending, EU), the DADVSI (passed, France), or some other legislation targeting copyright infringers, terrorists and paedophiles (but most likely the first). Oh and you americans, you're not safe either: they've already tried for mandatory trusted computing once. Opennet is a transitional phase, a means to an end: Your node will automatically reduce the number of opennet peers (Strangers) it uses as you gain new darknet peers (Friends) (thanks to somebody on Frost for the idea). In the long run there will be no opennet. But in the short run it should allow much faster growth and a much easier install for most people trying Freenet.

Apart from that, there has been some other progress: JFK has been merged (new connection crypto setup, strongly resistant to denial of service attacks), there have been many bugfixes including some fairly juicy ones (continually polling USKs, node stopping sending requests due to stuck inserts, etc), better IP detection (we run the plugins in parallel as they can be slow), translation updates, major bugfixes and improvements to jSite (it is now able to insert large freesites without silently dropping files after the 1000th), XMLLibrarian, the freesite search engine, is working and has a default index (type XMLLibrarian# or XMLLibrarian* into the plugins box to load it via HTTP; the latter will reload it on every startup), you can now tell a darknet peer to drop you when you remove it, datastore performance fixes, faster HTML filtering, cruft removal, and much more, all since the last update.

But on the whole, progress is relatively slow. Why? Partly that life keeps getting in the way for me (don't worry about your donations, I charge FPI by the hour!), and largely because we have relatively few volunteer devs at the moment, although we just gained an anonymous freemail dev from Frost (Freemail is now a plugin, and appears to work at least sometimes: install it by loading Freemail* from the plugins page [ will reload from HTTP on every startup ], and send me a freemail!). Freenet is really a much bigger undertaking than the current team can handle in a reasonable time. If you want to help build a critical piece of a free online future, please contact us on IRC, the devl lists, or Frost. If you're interested in becoming our second full-time paid dev for Freenet, that might just be possible (no promises, sorry...), I advise you to start coding and contact us once you've established a reputation for being a capable coder (after say a month); that's what I did.

And the Google Summer of Code 2007 is well and truly over, I've sent a final round-up to the development list; we learned a lot from this year, and that was cemented by and added to at the conference Google hosted for us (which originally I went to because I didn't want nextgens to be the sole voice representing us! It was great though, much better than last year). Many thanks to Google for SoC, even though many of our students were disappointing; we got some useful code out of it, and at least two of the devs will hopefully stay on. Next year, if we are accepted, we will do things a little different!

Expect the next update when you see it!

2007/09/06

Something happening?

Google Summer of Code is finally over. Yay! Several of our students are likely to stay on as devs, and some useful code has come of it, although most of the projects were not fully achieved. Rough summary:

• Swati Goyal worked as my student on the XMLSpider and XMLLibrarian plugins. While there remains a lot of work to be done for these to be really good, both now work reasonably well. You can spider the freeweb with the spider, it will produce an index in myindex7/ every 10 minutes; copy that to a separate directory, insert it as a freesite. Then feed the URL you get into XMLLibrarian, and you can keyword search the Freesites it found. For obvious reasons the Freenet Project cannot maintain an official index, hopefully several anonymous persons will maintain indexes. You can group multiple indexes into a folder. Eventually we will have embedded search forms in freesites.
• Mladen Kolar worked as my student to develop a comprehensive yet simple to use FCP library in C++ using Boost - a library to make it easy for C++ developers to write Freenet clients. This seems to work, but it needs documentation and better example code; hopefully this will be dealt with soon. It may also eventually be wrapped in other languages.
• Vilhelm Verendel has been working on simulations over the summer. He's one of Oskar's students, and has provided valuable input on various aspects of routing and swapping. Most recently, after a paper confirmed what we had suspected for a while, vive's work enabled us to tweak the swapping algorithm to prevent the network degenerating into a state where the entire network is clustered around a small number of locations, either due to churn or malicious attack (some believe we have recently been attacked on this level...).
• Frederic Rechtenstein has built Echo, a blogging wizard/plugin for Freenet. I understand that this is very close to being usable, but it has some packaging problems. The objective is to make it as easy to create a blog on Freenet, in a completely decentralised way, under your control, as through any of the major commercial blog hosts.
• Alberto Bacchelli has been working on unit tests for many of the support classes that Freenet depends on. In the process he has found a number of bugs.
• Srivatsan Ravi has been building new link layer encryption code, which will be resistant to denial of service attacks. This will be merged when it is ready.

Apologies for the limited progress with Freenet recently: I never seem to finish anything, life has been rather mad lately, it'll be sorted out soon (after we move)... Current priority items:

• Opennet - Universal Plug and Play support for the opennet port, allow reconnection for a short period after downtime, announcements - we need to finish opennet ASAP, this will hopefully reduce the number of people who leave because they have to perpetually chase noderefs, although it's by no means a long term solution: you will all need true darknet peers eventually, because they will come and harvest the opennet and render you unto the people with the unusual definitions of torture (he resigned, yes!).
• Ultra-Lightweight Passive Requests - necessary for outbox polling (spam resistant) Frost, useful in general... started but not finished, current priority depends on whether progress is made on Frost...
• LRU not preserved on datastore recovery - we need to fix this, it causes loss of content (SSKs are lost too :( ).
• Probe traces - we are using this network-level debugging tool to explore the network topology and prototype new routing mechanisms, specifically the rabbit hole avoidance protocol. At the moment it isn't working. Arguably the rabbit hole avoidance isn't needed on a true opennet - but a hybrid network is bound to have pockets of nodes, dungeons, which are difficult to get out of once entered, and this is why we need to consider how to deal with this.
• SSK request flooding - it seems that we send a lot of SSK requests for ARKs and so on, to the point that the spider doesn't work if its priority is too low. This needs investigating.

Finally, I wasn't sure whether to post the following: some folk might get the wrong idea, and it does break my quasi-anonymous persona somewhat (not that I'm actually anonymous, of course..). But I promised to host it, so you'd have found it eventually, and more importantly, it might do some good: ritual humiliation (completely unrelated to Freenet (on the web).

2007/06/14

1037

Upgrade to 1037! Really! The long-awaited Update over Mandatory support will make life much easier for the developers when debugging the network, because nobody will be left behind by a mandatory build ever again (well at least until the next incompatible crypto/packet level change), and the new probe request code will allow us to test a new backtracking/HTL algorithm (which should help significantly on darknets and hybrid darknet/opennets) without breaking the network. Lots of other goodies too (less short packets, possibly less timeout, f2f file transfer...). And it'll be mandatory on tuesday. So go get it!

One reason for the short mandatory is that work on opennet will start in the very near future... We need opennet, because of topology issues (#freenet-refs sucks), and because we need more users, more developers, more money, more content. And we need it to grow in order to have enough users to be able to build a global darknet during the limited time remaining to us. So watch this space!

One last thing. On the resources page, you will find a full set of debian AMD64 ISO's (all 21 CD's). Let me know if you manage to fetch them! (Or even one of them...)

2007/05/25

Moving along

The original plan was to capture a load of data about the network topology and use it to determine what is wrong with the network and whether opennet is The Ultimate Solution™ (or just a necessary evil). In the process we discovered a routing bug, which was probably causing the topology data to be corrupt, as well as various more serious issues, and is fixed in 1033. It will be mandatory on June 5th, so that gives a window to fix some more bugs and implement some long-planned features in the meantime (not necessarily all of these todo items):

• Binary blobs - migration of your favourite freesites between disconnected darknet without knowing the privkeys. Required for next item. 75% complete as of this update, should be in 1035. Less than 1 day.
• Update over mandatory - updating your node from your peers, even if your node is so far out of date that it can't route requests to them. Will enable faster mandatory upgrade cycles when we are deploying new routing, load balancing code etc. 1-3 days.
• Datastore fixes - various bugfixes, but the main thing will be including all the data needed to reconstruct the index in the store files (right now if the index is corrupted we lose all SSK data and the LRU list). 1-2 days.
• New plugins - since several of our Summer of Code students will be using plugins, and since several third party projects are already using plugins, it's about time to sort out the mess that is the Freenet plugin API!
• New congestion control - Michael Rogers has built us a new transport layer architecture, thanks to extensive simulations and input from related protocols. This involves a new packet format and new congestion control code. And perhaps even a new load management layer (token passing).
• Multi-container freesite support - This must be implemented before releasing Freenet 0.7 IMHO. It would greatly improve the performance of larger freesites (both inserting and requesting). tar.bz2 support would improve it even more, but the only java version of tar I know about, Apache Commons Compress, has licensing issues (it's incompatible with both GPL2 and GPL3).
• Insert memory usage - Early this year a load of work was done on reducing the memory footprint of large requests. Similar changes haven't yet been done for inserts.

PS: Move your freesites! Build 1035, out Real Soon Now, will by default disable access to pre-1010 insecure SSKs (you can still enable access manually). In a month or two we will drop the back compatibility code which allows these (and old CHKs too) to be stored. What this means is you need to move your freesite now if it is still using an insecure SSK/USK key (one which ends in BAAE rather than CAAE, and for CHKs AAEC* rather than AAIC*). Or move your favourite unmaintained site (suggest coordination via Frost). I have moved the Greater Toad Pictures Stash.

One last thing: The first 11 CDs of Etch for AMD64 have been inserted on the resources page. Let me know if you manage to fetch any!

2007/04/28

Looking up for once... mostly!

For once things seem to be looking up. The network is, compared to recent history, zooming along, inserting a CD image in around 2 days, routing is highly specialised even on low bandwidth nodes, link crypto is secure (thanks to STS), we have 6 SoC students and I only have to mentor two of them, and FPI is in reasonable health due to Ian's newfound talent for finding rich donors. The next build will feature a full localisation infrastructure (this is 99% done, I just need to put a few more l10n keys in), so hopefully in the near future Freenet's web interface will work in your native language (contact us if you are a native speaker of a non-english language but also speak good english and want to translate for us, or validate other people's translations). As regards opennet, we are currently collecting data on the network topology, which should tell us whether #freenet-refs is having a major negative impact; if it is, we will commence work on opennet in the near future.

Now for the bad news. IPRED2 passed the European Parliament. This piece of legislation makes inciting, aiding and abetting, attempting or doing any intellectual property infringement intentionally (not defined) on a commercial scale (loosely defined) a criminal offence subject to full criminal sanctions (prison time, judicial winding up, denial of legal aid etc). In its original form this applied to all "intellectual property", which would probably have criminalised the entire EU software industry; fortunately the passed version excludes patents. Unfortunately, it would probably still make Freenet illegal under incitement or aiding and abetting. It is likely to be some time before this completes the EU process, and once it does it can be up to 18 months for it to be enacted in each state's laws. So we are on borrowed time. If you want to help Freenet out, now's the time!

In unrelated news, check out Ripple, it's a friend-to-friend darknet-style currency system (there is an old version which is fairly centralised but there is slow progress towards a true f2f system, I've had interesting discussions about routing with Ryan). The basic principle is that money is a system of IOUs: it's information plus trust. So make that system explicit, and instead of trusting the bank all the time, keep it as close to your social network as possible. It would be quite interesting for LETS style hour currencies, but even for hard cash it would be useful. Have a look.

2006/09/18

Crash

Backups are important. Not only that, but backups of the right files are important! I lost a hard disk... it turns out that my (weekly) backups didn't include my most recent GPG key (but I have the old ones), nor my Frost identity. They do now! On the upside, Servalan (aka amphibian.dyndns.org) now runs Debian Etch on amd64 (the workstation still runs 32-bit, to minimize disruption and because of openoffice and gaming complications). Now, how do you know I am still me?:

• I still have the keys to this freesite.
• I still control toad_ on irc.freenode.net (a registered nick with op rights)
• I have a page about this on amphibian.dyndns.org.
• I have signed the message announcing this with two older keys which you can find on the public keyservers, and with which I probably signed some messages on the lists in the distant past: here and here.

My new public key is E43DA450 in my keyring (4096-bit, yay!). The crash message is signed here with the new key. My new Frost ID is toad@zceUWxlSaHLmvEMnbr4RHnVfehA and my new freemail address is toad@freetoad.freemail (long version), although this isn't necessarily fully working/integrated yet; don't send any important correspondance over it yet, but if you want to send me a test message I'd appreciate it.

Geek computers are like the TARDIS. They are immensely powerful, but they're constantly breaking, because we're constantly tinkering with them! (Well in this case it was a case of abusing commodity hardware; why should a hard disk fail? I know even Seagate drives have a limited lifespan, but they shouldn't! :)).

On the other hand, last week's holiday (Monday to Thursday) was pretty cool. Me and mum went to see granny in Harrogate. Spent some time with granny, visited some nice green places, got lots of cheap videos from the charity shop, watched all four Harry Potter films, discovered that a cousin of mine is a Red Dwarf fan. Normal service (normal chaos?) will be resumed shortly; most of the TODO list on the last but one item still stands :(.

PS, here is the text of the Pope's original speech, if anyone cares; it's primarily about religion and rationality, but does contain a subtle attack on Islam even if we accept his apology with respect to the most blunt quote.

PPS, the below item (Hereticnet) has a Frost board called Hereticnet.

2006/08/29

Heretic

There are a number of persecuted groups who would greatly benefit from freenet's technology, but who cannot use it for moral or political reasons. For example, persecuted churches. Even if you are an atheist I hope you accept that freedom of thought, and therefore of religion, is important: You have the right to sincerely believe in the Jesus, Buddha or the Flying Spaghetti Monster, if you want to. Others have the right to ignore you and think you are crazy. So there may be room for a darknet variant which uses a lot of freenet's code, but has different goals. Note that I am saying nothing against Freenet itself: I like Freenet, I am morally happy with it, but I think there may be room for something else as well, once Freenet has reached a reasonable level of stability.

Such a network would be resistant to external censorship, but provide for internal censorship. In other words, it would be a high standards darknet: A community with its own standards for content, which it could enforce through expulsions and schisms, but which is not necessarily the same as the outside world's standard. On such a network, content inserts would be tagged with a cryptographic structure allowing the insert to be traced back one hop at a time, but only with the consent of (for example) 2/3rds of the nodes surrounding each hop. If somebody found some content they object to, they could file a complaint. This would be discussed on the chat system, and ultimately people on the network would inspect the disputed content (hence the need for a fairly 'high' standard), and decide whether to vote to trace the author, to trace the complainant, or to do nothing. If enough nodes vote to trace the author at each hop, he would be traced. He would then be identified to his direct peers, and everyone else would know his topological position. The network must then decide what to do with him. His direct peers may simply disconnect from him. Or they may choose to protect him, (either after the trace or during it), in which case they themselves may be disconnected from. Irreconcilable differences will have to be dealt with by a larger network split: What was one community is now two.

This is by no means an easy way out of the conundrum that is freedom of speech. It requires significant effort on the part of the users, and it also requires a fairly high standard; "anything but child porn", for example, is likely to result in permanent brain damage (or at least a need for counselling) to active participants on the network, since disputed content will normally be close to the border between what is allowed and what is not. A persecuted church would have a much higher standard, while most of its content would still be illegal by the local laws. And it is likely that such networks would have major problems with splits and schisms, as any other community does. It would closely represent the underlying community. It seems to me that this would be an interesting experiment, and it might be useful to somebody. Comments? Contact me!

2006/08/19

Rolling on

Probably a good idea to update my freesite once in a blue moon, right? Well there have been lots of goings on in Freenet land... Minor stuff first. Build 950 somewhat experimentally makes the node treat backoff as advisory rather than mandatory, so your node will send requests to backed off nodes if it can't send them to nodes which aren't. This seems reasonable to me, (because we have a separate load limiting system), Anyway it's not a long term solution; the current mechanism has major security issues (flooding and possibly some local attacks too). Tell me what you think of performance on Frost. There have been many other improvements, a good deal of which is due to our volunteers and Google SoC students (students Google pays to hack on freenet over the summer); upgrade! As you've probably heard, John Gilmore (who remarkably enough owns toad.com) has donated $15,000 to Freenet, so I'm definitely going to be paid for the next six months. Obviously this is a good thing!

One of the big questions in 0.7 now is opennet. Here's the deal. On darknet, you only connect to your friends (in theory; in practice most links are established via #freenet-refs etc between strangers). On opennet, the network assigns you connections. Either way, you are vulnerable to your peers: they can do correlation attacks, and probably lots more fun things. On opennet, you are probably connected to the Bad Guys, because they probably harvest the network (impossible on darknet), pretend to be a thousand nodes (hard on darknet), and connect to everyone they can - preferably more than once under different identities. In other words, Opennet is vulnerable to harvesting (quickly finding large numbers of nodes) and Sybil attacks. For the time being there isn't much we can do about correlation attacks; I have a few ideas which may be implemented soon but they won't provide a very large anonymity set; expect major progress in 0.8, but it's quite likely that premix routing won't work on opennet. So as you can see, we need opennet for bootstrapping: for getting slashdotters onto the network. In the long term, we need as many true darknet connections as possible, because Freenet will be illegal or attacked. One thing which is important is not to jump in too soon; many problems are easier to solve on darknet, which has lower connection churn etc. But we do need opennet, and once we have it we need people to move from all opennet to some opennet and some true darknet connections, and eventually to pure darknet, or the network will be unsustainable. What incentives we can provide is the subject of some debate.

So when will Freenet be illegal? The DADVSI in France arguably makes it illegal there. We may have a constitutional case, or we may not, but from what I've heard we are in trouble. People who distribute or develop it may be liable to prison time and a €300,000 fine. What's more fun than the DADVSI then? The IPRED2 is a directive going through the European Parliament which makes it a crime, punishable by all manner of criminal sanctions including prison time, judicial winding up, large fines, and lots of other fun things, to intentionally commit, attempt, aid, abet, or incite (!) the violation of any intellectual property on a commercial scale. "On a commercial scale" is undefined, but caselaw suggests it's not a real protection for anything either Freenet or the free software movement does; "intentional" may not require knowledge of the specific IP according to the FFII. But even if it does, knowingly violating a software patent on the grounds that it's a load of nonsense would now become a criminal offence! The application to patents alone will seriously harm the free software movement, of which Freenet is a part, and fighting it on the patents issue is the best way to attack the directive (as we won the fight last year). But the application to copyrights might well ban Freenet EU-wide. Please don't get depressed and cynical about this. Join the FFII, read as much as you can, and write to your MEPs. Last year we fought the proponents of software patents (including IBM, which is either schizophrenic or trying to have its cake and eat it, or both) to a stalemate: All that is required for evil to triumph is that good men do nothing, but there is much that we can do (before we reach the "we must all hang together..." stage!). Here's a letter (HTML) I sent recently (addressed to Lord Sainsbury, although you should generally write to your MEPs on this); please don't send exactly the same letter, but you're more than welcome to use it in writing your own.

Finally, the government expects all UK broadband ISPs to be blocking child porn sites using Cleanfeed by the end of 2007. They have assured me that it will only be used for this purpose, but I still worry that ISPs will be forced by litigation to block specific copyrighted content (e.g. xenu.org) - certainly they will if the IPRED2 passes. We shall see...

TODO LIST: Current relatively high priority items:

• Multi-container freesite support - currently any content in a freesite over the 2MB container limit is inserted separately; multi-container support will help a lot with big sites.
• Station-to-station protocol - Freenet 0.7's crypto at the moment is vulnerable to both spoofing and MITM.
• Low level rewrite - Michael Rogers and I have been debating congestion control, bandwidth limiting, load limiting, and a little bit of encryption for some time, while he has been writing simulations of these for his Google SoC project. A lot of this could be put into practice in the near future, although it'll be a while before token passing load limiting is ready (but when it is it will solve lots of problems).
• Location probe requests - We have some persistent suspicions that location swapping is causing large numbers of nodes to have locations in roughly the same place, and sparse keyspace regions populated only by newbie nodes which don't last long. Location probe requests would let us investigate this, and also get an accurate size estimate for the network.
• Freemail and SNMP - SNMP has been working for some time, and I want some pretty graphs! Freemail (Dave Baker's Google SoC project) is now entering final testing. Hopefully next time I post I can include a freemail address!
• Other security issues - There are some nasty security tradeoffs in 0.7 (e.g. whether to cache local requests), I have some ideas for improving security without too much work (the big gains are in 0.8 where we get premix routing etc).
• General bugfixing etc - Many minor but important features aren't yet implemented, and there are always more bugs...
• And of course, OPENNET !

2006/06/02

Shadows of shadows

A freesite with a few links to stuff downloaded from a certain p2p program recently came to my attention. It contains at least one file which is pure copyright infringement, so I won't link it here. But it also contains this file, labelled on the original site "US ARMY 3 - top secret defense - Biological Weapons Technology.pdf". Sorry to burst your bubble, but it's on the web. Just google for "Those delivering BW could be". That gets you to www.fas.org/irp/threat/mctl98-2/p2sec03.pdf. Go up a few levels and you find all sorts of wonderful stuff about WMD and intelligence threats and so on. Another few clicks and you get to stinet.dtic.mil. An amazing amount of this stuff really is just out there, and there is absolutely no shame in pruning through it and finding the good stuff. Or even calling attention to it. I honestly don't know what the legal situation is with this sort of stuff, but frankly I don't care; it's almost certainly covered by copyright exceptions in my country. Have fun, and insert anything you find especially interesting!

Welcome to Freenet 0.7!

I have finally updated my Freesite, two years after the last post - and this time it's on the long-promised Freenet 0.7 alpha darknet! Yay! here is a page about Freenet 0.7's current security or lack thereof, from the wiki. Also, we are looking for student coders to work with us for money over the summer, thanks to Google's summer of code. You will however need to blow your anonymity by visiting the web site to find out about this.

Naphtala

The excellent (if small as yet) new index called FreeHoo carries in-depth reviews of Freesites. His second site listed was Naphtala, a site which I have never visited because I believed it to contain child porn. It doesn't. It contains a fair amount of interesting content written by a paedophile, none of which qualifies as pornographic or illegal as far as I can see. He claims at one point to use non-violent child porn so that he doesn't have sex with real girls, while consistently defending his practice of "child love". Definitely worth a read, as Freesites go, but I won't link it here directly for purely paranoid reasons. Obviously I do not endorse his views: any sexual relationship with a child is evil, illegal, a sin, and likely to damage them for life (I go much further than most people here by endorsing the biblical view of sex, which is that it is a good thing, within marriage). However it is interesting to hear from the other side, and I rather pity Naphtala.

Brain Dump - why do I do Freenet?

I was recently bothered by the whole thing of why I do Freenet, and whether it is compatible with my ethical basis. I include the resulting braindump in case anyone cares. You may find it interesting. I rather hope this won't result in me becoming even more of a cult figure, but perhaps it will spread some useful insight.

Why do I think it is right for me to do Freenet?

As an evangelical Christian, I'm not the stereotypical Freenet user. I don't surf the porn sites, I accept an ultimate (although not human) Authority, and so on. I don't define myself as a Libertarian or Anarchist. Many of my brothers and sisters in Christ would probably not approve of what I am doing. So, why do I think it is appropriate for me to do Freenet?

Firstly, Why is this difficult? Well, firstly, my income comes from working on Freenet. I don't have any real financial commitments to worry about, but it is better that I am able to contribute to the family running costs. And I enjoy giving to both Tear Fund (an explicitly Christian overseas charity operating in Africa, Columbia and other "nice" places; they help people to work their way out of poverty, provide emergency relief, and witness the gospel in both their words and actions) and my local church. Thus I have a vested interest here, which makes objective ethical decisions difficult.

So, now the meat. I do Freenet because I believe that freedom of speech is, despite not being a biblical concept, something worth fighting for in a democracy. Does this mean I idolize it? If the state has the right to taxes ("give to caesar what is caesar's...") and even to wield the sword (one of Paul's letters), then isn't it legitimate for it to say what people can and can't say? It is only a small step from here to "Isn't it legitimate for the state to say what people can and can't believe? Isn't it within the bounds of statehood for Rome to persecute the early Christians?". If we look at it from this perspective... Rome persecuted the early church, and failed. Badly. The church did not fight Rome (it was not a good idea historically, nor was it compatible with the sermon on the mount). But the Church prevailed, and those martyred (in horrible ways) will be rewarded in heaven for their perseverence.

So how does this relate to Freenet? Well, Freenet can be used for good or evil; it could be used by Chinese student groups to keep in contact with each other and the West (for example to organize prayer when they are particularly persecuted), by urban Saudi churches to keep in contact with current Western christian literature, by Western whistleblowers to publish things such as the Diebold files, or to publish criticism of the Church of Scientology, by paedophiles to exchange pictures of child abuse, by others to publish racist hate speech, and by yet others to publish lists of people accused (but never proved.. or who have served their time) of paedophilia and related offences, for future ostracisation and lynchings. Surely it is the state's right to prevent the last 3 cases?

Firstly, even if there was no Freenet, there would still be files that while technically illegal are clearly in the public interest, and should be hosted by as many people as possible. For example, the various sites on the Church of Scientology (read them, they're on Freenet). These are of clear benefit to the kingdom of heaven. The state's regulatory role is trumped by the clear good that is being done in keeping people away from this vile cult.

I have always thought that what Paul said about the state's right to wield arms, and so on, are guidelines - true in general, but occasionally it may be necessary to go up against the state; to disobey, in extremis to actively work against the state, perhaps using violence as a last resort. The resistance against the Nazis in WWII is one clear example of this. If you have Jews in a cellar, and the gestapo come and ask you about them, it is perfectly okay to lie to them. If a higher cause is being served it may sometimes be necessary to steal or even to kill, in such extreme circumstances. However on the whole, people lie to cover up sin, or to "fit in" (fear of man"), people steal to feed their sinful desires, and people kill for much the same reason.

So where does that leave us? We have to fend for ourselves, and work out exactly what the costs and benefits are. Even if the state has a legitimate right to suppress lists of people accused of crimes they were never prosecuted for (for obvious reasons), that does not mean that it will be successful. Very often such people are persecuted - often in error - because somebody in the police themselves tipped off the locals, or the paper. Once this information is available, it will spread, because it is believed to be in the public interest. Sometimes perhaps it is. If such data is published on Freenet, and the author is believed (which may be difficult - he needs to earn some trust - but this is certainly not impossible - and in this particular case it may not be hard :( ), then it is difficult for the state to get rid of it. To that extent, our building of Freenet has helped this to happen, even if we do not ourselves approve of such content's publication. Similarly in the case of paedophilia: we hide the people distributing the images, and although the police may well be able to investigate in other ways (e.g. matching the children involved), they are not able to simply trace the distributor, unless he does something stupid to give away his identity (trapping him on a message board may work). They are therefore available to a slightly larger audience at a somewhat reduced risk (although theoretically if they are not accessed they SHOULD fall out of Freenet).

So we are clearly building something that CAN be abused. As can a kitchen knife, or just about anything else, but more specifically, weaponry. We are building a weapon here, of a sort. Like all weapons it is not tactically neutral; it cannot be used to suppress information. It can be used to prevent such suppression. My intent is that it be used for the positive purposes listed above, and more. But some will inevitably use it for evil. How do I know that the good will outweigh the evil? Lets look at the good first:

Freenet can probably be shut down relatively easily in its current state by a moderately well funded attacker. For example, a half time Chinese technician (of some skill), with access to the firewall rules, could set up some nodes, harvest some references, and also the seednodes, and block every node found by IP address. However, compare it with what they have now (or did before recent developments): a mailing list on which open proxies are announced. Every 24 hours or less, a new open proxy is found and announced. The government is subscribed to that list, and within 24 hours, the new proxy has been blocked. Using Freenet (or I2P, which has similar attack issues) instead makes the cost to the Chinese government higher. If Freenet gets bigger, it becomes harder for them to block, but it remains reasonably feasible. However, should Freenet routing start to work, we may be able to provide new features that make it substantially more difficult: trusted mesh routing, which would severely limit harvesting in hostile environments, and thus limit the damage from busting one node; new (steganographic) transports, which slow down and make more difficult attacks, and so on. In fact, right now, you can identify all Freenet traffic by some identifier bytes. But we will get rid of this. And even if Freenet is never really THAT useful, there may well be future anonymous P2Ps that are. We will have laid the groundwork, even if they don't use any of our code, through our extensive practical research. Little of it is formally published, but I still think a lot of that knowledge would be out there.

In terms of impact: revolutions are always (in China anyway) led by the students. Even in China, the students have internet access. Bloodless revolutions have happened on many occasions in the last century. But the people need to know that another world is possible in order to demand change. I do think that such things *ARE* God-blessed. Velvet revolutions as we saw in the Soviet Union, in Serbia, in Georgia, are usually surrounded by a delicate chain of coincidences that is the signature of divine intervention; God *is* interested, and he *does* care. And the stakes are pretty high: there is a good chance of China and the USA having another cold war, quite apart from human rights issues, and the persecution of the Church in China.

In the West, files such as the Diebold files (leaked info on voting machines with major security issues), the Church of Scientology files (which the Co$ has consistently abused copyright law to pursue, even if they are readily available *NOW*), and so on, demonstrate that freedom of publication is vital to protecting democracy. And democracy has achieved great things for us, even if it is constantly under attack. At its best it can be a great means for change in a positive direction. At its worst, it devolves into plutocracy, as do all other governmental systems. The basic difference is the attitude of the electorate. If they are apathetic, evil triumphs; "All that is required for evil to triumph is that good men do nothing", as a wise american once said. So as always, true progress comes from the renewing of minds; but nonetheless democracy is valuable and worthwhile. And without freedom of speech, democracies devolve, as the public do not know what is going on, even if they DO care.

In the future, the West may become a very dark place. I cannot know for certain but I suspect we are in the twilight of democracy here. The War on Terror expands, and if it is not over quickly, it risks destroying democracy. If there is another serious attack, we will take another major step into the abyss. The intelligence war may be going well, however, there is plenty that the terrorists could do, and the war for hearts and minds is largely ignored (as we see with the reluctance of the US to bring real pressure to bear on Israel). If it continues (and it may, as there is considerable money to be made on both sides), we are going down the tubes - and this can't easily be fought with the prospect of terrorists crashing jets into nuclear reactors on the horizon. The other big influcence is in cyberspace, which increasingly dominates meatspace as a communications medium. It is quite possible that the little war between open source software and Microsoft et al will come to an end. If Microsoft loses, its stock price will collapse, and this will have substantial effects on the US economy (IIRC Microsoft is 20% of the S&P index, which a lot of pensions are linked to; not sure about the NASDAQ). If Microsoft wins, we will have increasingly unreasonable DRM regimes, and the War on Piracy will start to move. Because there are always bugs, automatic updates of trusted operating systems are inevitable. Because you cannot prevent people from aiming a camera at a TV screen, we may well be looking at a means of remotely deleting files from all trusted PCs everywhere that match a particular checksum. Once this is technically available, even if judicially supervised, it will be abused. Eventually it will be abused for overt political goals, but where there is a blur between copyright and freedom of speech, the former will prevail, as it usually has in the past, and increasingly will in the future as further legislation passes which makes it ever more obvious which side the legislators' bread is buttered on. Freenet will not be the only key piece in this fight; black-market hardware may well be an issue, and there will be many more; and I personally will not be involved, once OSS (or Freenet) is made illegal, because of my prominence beforehand; but this is another area where Freenet and similar technologies may well be of some practical benefit. And the Church will not be unharmed either. Prohibiting radical islam would be politically incorrect, so what is likely to happen would be prohibiting religious intolerance (there are already laws about this, we will have to wait and see how they are interpreted). Sermons and literature on why other religions are wrong, or why homosexuality is wrong, and so on, would become risky. If the WoT continues to escalate, or if there is a regional nuclear war (as the world continues to "tool up"), we may end up with explicit regulation of religion.

So, in conclusion: I do not know whether Freenet will be a net benefit. I do know that I am writing a tool that can be used for good or evil, that the good it can be used for is considerable, and the evil that it can be used for is there. God, however, knows. So it all comes down to whether God is happy about it. Which is expressed in my conscience; the indwelling Holy Spirit informs a believer's conscience. And I don't really have a problem here, most of the time, despite having a better relationship with God than I had some years ago. It is also expressed practically. I pray to God for success, and I have success, often from unexpected quarters. I have had huge last minute donations; I have been led to enormous bugs by coincidence and hunches after prayer. I have been blessed in many ways. I also honestly believe that the project we are undertaking is massively difficult, and a lot comes down to luck; we do not have the resources for it to be otherwise. I don't believe in luck; I believe in providence. I have met several Christians through Freenet, despite the natural expectation that everyone involved in such a project would be either very shallow or a committed Libertarian atheist. And so I continue. I believe, and trust, that if it is wrong for me to do Freenet, God will reveal this, in time. And even if he does not, or if he makes it abundantly clear and I ignore it for years, I will be forgiven at the end of time, because He has paid all my debts, and gone where I could not return from, and come back, and given me the promise of eternal life, secured by the only true authority in the Universe. Glory be to the Lamb forever and ever!

Hutton Craziness An interesting puzzle...

The Hutton Report (Freenet) was published today. I haven't read it (been far too busy with Freenet), but the general gist of it, according to the press, is that the government has been completely exonerated. A lot of other interesting stuff came out during the process of the inquiry, but the conclusion is interesting, given the fact that we still haven't found any WMD in Iraq, and Hutton says the dossier (Freenet) was _not_ 'sexed up', we have a few options:

1. Hutton is wrong This seems to me to be the most credible option: the government did indeed produce a misleading bunch of factoids, either completely independant of any actual intelligence, or exaggerated and carefully selected to look good. This could be a more subtle long term problem: agencies, individuals, sources that produce good intelligence are rewarded, but there is no way to measure how "good" intelligence is, so you check it against your political goals and preconceptions...
2. There is a pig flying the plane If Hutton is right, then the intelligence services did indeed conclude that Iraq had Weapons of Mass Destruction, without prompting from the government. If this is the case, given that we haven't found any, we have some more options:
   1. Iraq really does have WMD We just haven't found it yet. This seems increasingly unlikely as time goes on. And if it does, they are highly unlikely to be deployable, since they weren't used in the war, and they haven't been found since despite the disbanding of the Iraqi army and totally unrestricted access for the team of 1,400 US investigators to all sites in Iraq.
   2. Our intelligence services are incompetent If Iraq does not have WMD, but the intelligence services believed impartially that it does, then logically a third world nation must have deceived the CIA, the NSA, GCHQ and SIS, DIS, and all the other three letter agencies involved in many countries. The US intelligence budget is rumoured to have been $28,000,000,000 in the late 90s and is undoubtely much higher now.
   3. Our intelligence services are malicious The other option is that they did not seriously think Iraq had WMD. If they then lied to the government, as opposed to the government lying to us, then they are completely out of control, and presumably had their own geo-political agenda, perhaps something about the 7 million barrels a day Iraq could produce if its oil reserves were properly explored, and what that could do for western economies. Or something less subtle... Of course this doesn't have to be a global conspiracy - presumably if the american agencies wanted to believe in Iraq having WMD, they could persuade the UK agencies of this fiction. Of course, if this is true, sooner or later something may be planted to sate the public...

The other interesting part is that no British government is going to go to launch a preemptive war on the basis of intelligence for the foreseeable future. This is probably the real payoff here. America loses its biggest ally in the fight against States We Don't Like. How sad. Sidenote: I actually think Iraq is better off for the war for humanitarian reasons. That's not really the issue here though.

And now for something completely different: Freenet Update Freenet is currently having some major problems. Both branches are running NGRouting, which could probably be improved significantly; m0davis is working on that, when he's around, which is rare. The unstable network probably has 100 nodes, and seems to work remarkably well for new content, with insert/retrieve tests succeeding immediately even at relatively low HTLs, most of the time. Stable however isn't. Stable is estimated to have on the order of 10,000 nodes, as of a few months ago: Iakin ran an ubernode for 4 days and had 16,000 unique IPs contact it. The maximum HTL has been reduced to 10 on the stable network to try to reduce load. At least on stable, nodes are in overload almost all the time, and it's almost all the time caused by bandwidth usage. However, this is expected; we now reject queries based on outbound bandwidth usage, with the result that when we come out of overload, if we get a few queries for files that are actually retrievable (and yes, this does happen), our bandwidth gets used up and we go back into overload. After a somewhat paranoid conversation with an old friend, I discovered a possible attack that could have caused the current symptoms, you can read the thread here, which was fixed in 5064, but not completely resolved, as you can read in the thread. This may be happening a little by accident, but I doubt that it's happening a lot by accident, because the log messages that would show it are relatively rare ("Got really late DataReply" is the obvious one). Anyway, 5064 fixes it, but it also makes probing the network for a given key a bit easier, so there will need to be further action on it. I am currently engaged in implementing a new load balancing system, based on the idea of enforced, explicit maximum request rates, which was originally proposed by Ian. The result should be that the load is moved back to the flooder - if a node makes too many requests, it will get RNFs rather than overwhelming the network. This should help to balance load and heal routing... but we've all heard that before. It also gives us some interesting possibilities w.r.t. fairness - we can change the minimum request interval on a per node basis, so that we can "punish" nodes for bad behaviour (such as the attack above), or accept more requests from nodes whose queries are most likely to be successful. This should not take more than a week to implement, and it will require a network reset. After that, we are going to attempt to set up a "testnet", an expanded form of the old watchme network - a completely separate, non-anonymous network for testing routing, debugging, and so on. Unfortunately for it to be of any use we will need several hundred nodes to run it. But eventually it should give us a much better idea of what is going on.

Rants go here! I replied to jrand0m's last post on the list IIRC, but future ones will be replied to in both places. I'm sure I'll have other things to rant about occasionally.

Since jrand0m asked nicely, and since I knew he wouldn't mind me borrowing his template, I've decided to set up a rant site of my own.

I actually got a mail from a Chinese citizen on the support list today. He can't get to the web site, as it is blocked. Hopefully I can send him a ZIP. I still think anyone using Freenet in China is crazier than I am though - but as Ian often points out, what they used before was even worse.

Stats: